Dynamic display of video communication data

ABSTRACT

Computing devices may implement dynamic display of video communication data. Video communication data for a video communication may be received at a computing device where another application is currently displaying image data on an electronic display. A display location may be determined for the video communication data according to display attributes that are configured by the other application at runtime. Once determined, the video communication data may then be displayed in the determined location. In some embodiments, the video communication data may be integrated with other data displayed on the electronic display for the other application.

BACKGROUND

The advent of mobile computing combined with more powerful networkingcapabilities has changed the ways in which individuals communicate.Formerly, specialized networks had to be configured with particularequipment to provide different types of communication. Videoconferencing, for instance, relied upon specialized telephone networks,radio frequencies, or closed-circuit television systems to providetwo-way or multi-way video communications between users at different(and often fixed) locations. As broadband or other high-capacitynetworking technologies have allowed greater and more flexible access toa variety of different systems or devices, via the Internet forinstance, different types of communication have developed or evolved totake advantage of these more powerful networking capabilities. Forexample, instant messaging technologies may provide real-timetransmission of text messages over the Internet, allowing participantsto chat between two Internet-enabled devices.

Video conferencing has also evolved to facilitate video communicationbetween network-connected devices. Video communication data may betransmitted over networks, such as the Internet, to other participants,providing a video conference-like user experience, without being subjectto the limitations of older and more static communication technologies.However, as many devices, such as laptops, mobile phones, tabletcomputers, or other mobile computing devices now facilitate multipleprograms or functions operating simultaneously, video communications maybe subject to disruption, inattention, or competition for computingresources of the device performing video communication, dulling thereach or effectiveness of new video communication technologies.

SUMMARY

Various embodiments of dynamically displayed video communication dataare described herein. Video communications between two or moreparticipants may be facilitated based on the transmission of videocommunication data between the participants. However, other applicationsmay be currently executing at one or more of the participants that maybe controlling or displaying image data or video data on an electronicdisplay. Video communication data received as part of the videocommunication may then be displayed within a portion of electronicdisplay as may be configured by the currently executing application. Thecurrently executing application may configure one or more displayattributes at runtime which indicate the display location for videocommunication data. In some embodiments, the video communication datamay be blended into other image or video data of the currently executingapplication to produce a single video stream to be displayed. Variousinputs may be received for the video communication, such as pausing,resuming, or ending the video communication, via touch input in thedisplay location of the video communication data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate peer-managed video communications betweendifferent participants, according to some embodiments.

FIG. 2A is a block diagram illustrating dynamic display of videocommunication data at a participant of a video communication, accordingto some embodiments.

FIG. 2B illustrates a participant device displaying video communicationdata in an integration display mode for a video communication, accordingto some embodiments.

FIG. 2C illustrates a participant displaying video communication data ina full-screen display mode for a video communication, according to someembodiments.

FIG. 3 is a block diagram of a video communication module, according tosome embodiments.

FIG. 4 is a high-level flowchart illustrating methods and techniques fordynamic display of video communication data, according to someembodiments.

FIGS. 5-6C illustrate dynamically displayed video communication data inan electronic display of a participant device, according to someembodiments.

FIG. 7 is a high-level flowchart illustrating methods and techniques fordynamically displaying video communication data for multipleparticipants in a video communication, according to some embodiments.

FIG. 8 illustrates dynamically displayed video communication data in anelectronic display of a participant device for multiple participants ina video communication, according to some embodiments.

FIG. 9 is a high-level flowchart illustrating methods and techniques forhandling input for video communication data that is dynamicallydisplayed, according to some embodiments.

FIGS. 10A-10C illustrates examples of handling input for videocommunication data that is dynamically displayed, according to someembodiments.

FIG. 11 is a block diagram illustrating an exampletransmission/reception manager module for a video communication modulethat implements dynamic pause and resume for transmission of videocommunication data, according to some embodiments.

FIG. 12 is a high-level flowchart illustrating methods and techniquesfor dynamically detecting pause or resume events for a videocommunication, according to some embodiments.

FIG. 13 is a high-level flowchart illustrating methods and techniquesfor determining an active exchange indicator of a video communication,according to some embodiments.

FIG. 14 is a high-level flowchart illustrating methods and techniquesfor a dynamic transition from a video message to a video communication,according to some embodiments.

FIG. 15 is a high-level flowchart illustrating methods and techniquesfor monitoring the quality of video communication connections fordynamic transitions between video communications and video messages,according to some embodiments.

FIGS. 16A and 16B illustrate a dynamic transition for multipleparticipants in a video message to a video communication, according tosome embodiments.

FIG. 17 is a block diagram illustrating a connection manager of a videocommunication module implementing dynamic transitions between videocommunications and video messages, according to some embodiments.

FIG. 18 is a high-level flowchart illustrating methods and techniquesfor instant video communications, according to some embodiments.

FIGS. 19A and 19B are high-level flow charts illustrating methods andtechniques for obtaining authorization to establish instant videocommunication connections, according to some embodiments.

FIG. 20 is a high-level flow chart illustrating methods and techniquesfor display video communication data via an instant video communicationconnection, according to some embodiments.

FIG. 21 is a block diagram illustrating a connection manager of a videocommunication module that implements instant video communications,according to some embodiments.

FIGS. 22-24 are block diagrams illustrating an example portablemulti-function computing device, according to some embodiments.

FIG. 25 illustrates an example computing system according to someembodiments.

While the invention is described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that the invention is not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit the invention tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope of the present invention. The headings used herein arefor organizational purposes only and are not meant to be used to limitthe scope of the description. As used throughout this application, theword “may” is used in a permissive sense (i.e., meaning having thepotential to), rather than the mandatory sense (i.e., meaning must).Similarly, the words “include”, “including”, and “includes” meanincluding, but not limited to.

DETAILED DESCRIPTION

Various embodiments of dynamic display of video communications aredescribed herein. Video communications may occur between differentparticipant devices. These participant devices may capture video andaudio data (usually of a user) at the respective participant device andtransmit the collective video communication data to other participantsin the video communication for display. While video communication datamay be displayed under the control of a video communication application,for example in a full-screen display mode, other applications may beconcurrently executing at participant devices. These applications maycontrol or utilize some or all of the display for image data or othergraphics for the application. Dynamic display of video communicationsmay allow for the currently executing applications to control, direct,or otherwise configure where the video communication data received forthe video communication is displayed at the participant device.

Various embodiments of dynamic pause and resume for video communicationsare described herein. As noted above, other applications or tasks may beexecuting at a participant device in a video communication. At times,therefore, the level of participation or active exchange of informationvia the video communication may fluctuate. Evaluation of thecommunication data that is captured for the video communication mayindicate whether transmission of some or none of the video communicationdata is necessary. Dynamic pause and resume for video communications mayprovide an automated or intelligent detection of pause events, whereactive transmission of some or all of the video communication data maybe paused without prohibiting the exchange of information via the videocommunication among active participants. Similarly, an automated orintelligent detection of resume events may allow for the suspendedtransmission of video communication data to resume. Computing resourcesat the participant device may therefore be dynamically conserved.

Various embodiments of dynamic transitions from video messages to videocommunications are described herein. Video messaging is used to sendvideos to recipient devices that may be played back at the convenienceof the recipient. Exchanging information in this manner may be lessefficient for a user than multi-directional communication, such as videocommunication. Dynamic transitions from video messages may beimplemented such that if a user of a recipient device receives a videomessage and wishes to respond or engage in further communication, avideo communication may be established instead of replying with anothervideo message. In this way a user may respond to the video message withmore flexibility, either a video communication may be established, orthe video message may be saved for later play back and/or response.

Various embodiments of instant video communication connections aredescribed herein. For communications between certain users, it may bedesirable to prefer video communication over other communicationtechnologies, such as voice or text. Family members, friends,co-workers, or other communicative relationships may take advantage ofquick and efficient video communications. Instant video communicationsmay allow users to authorize pre-established video communicationconnections between respective participant devices, so that videocommunications may appear to begin instantly. For instance, a videocommunication may be displayed as soon as it is received, minimizing thetime to prior to engaging in video communication. Users may manage orauthorize which other users (or their respective participant devices)may establish instant video communications with their own respectiveparticipant device.

Video communications may occur between two or more participant devicesover a wide area network. Participant devices may implement some kind ofapplication, module, or component that may facilitate videocommunications, from establishing connections, to capturing,transmitting, and processing received video communication data.Different types of participant devices may establish or participate invideo communications, as well as different numbers of participants, suchas two-way or multi-way video communications. In various embodiments,video communications may be peer coordinated or managed. For example,participant devices engaged in a video communication may utilizepeer-to-peer communication strategies or protocols in order transmit andreceive communication data. Please note, that peer-managed videocommunications may not preclude the use of third-party systems orservices to facilitate portions of the video communication, such ascellular or other types of wireless network services that may provideaccess to a wide area network for participant devices, authentication oridentification services, or simply relay or retransmit messages or datato ensure that the video communication reaches all participants. In atleast some embodiments, peer-managed video communications may notinclude a centralized video communication coordinator, server, system,or service, but may instead allow the participant devices as peers tomanage or coordinate the transfer of video communication data among theparticipant devices.

FIGS. 1A and 1B illustrate peer-managed video communications betweendifferent participants, according to some embodiments. Participantdevices (e.g., participant devices 102, 112, 122, 132, 142, 152, and162) may be implemented on different devices or types of devices.Examples of participant devices may include, a portable-multifunctiondevice illustrated in FIGS. 22-24, such as multifunction device 3000,other types of mobile computing devices or phones, computers, laptops,personal digital assistants (PDAs), or any computing system, such ascomputing system 4000 described below with regard to FIG. 25. Respectivevideo communication modules (e.g., video communication modules 104, 114,124, 134, 144, 154, and 164) may be implemented at each participantdevice in order to facilitate video communications over wide areanetwork 100. Video communication modules may be implemented in manydifferent ways. FIG. 3, described in detail below, provides an exampleof some embodiments of a video communication module.

Different video communications may be established by video communicationconnections through wide area network 100 (e.g., the Internet). In FIG.1A, for instance, a two-way video communication may be establishedbetween participant device 102 and participant device 104. While in FIG.1B, a multi-way video communication may be established between amongparticipant devices 124, 134, 144, 154, and 164. Participant devices maybe configured to access wide area network 100 in different ways. Forexample, participant device 102 may be configured to access a particularwireless network, such as a cellular network, that may communicate withwide area network 100, by use of gateway. A gateway may provide a packetoriented mobile data service, such as General Packet Radio Service(GPRS), or other mobile data service allowing wireless networks totransmit data to other networks, such as wide area network 100.Likewise, different access devices or points (e.g., IEEE 802.11gwireless access router or device) can provide communication access towide area network 100. Participant device 112 may, for instance, beconfigured to (or authorized to) utilize the access devices to obtainaccess to wide area network 100. Thus, in various embodiments,participant devices can access wide area network 100 in different waysand still establish video communications. Thus, the previous examplesare not intended to be limiting as to other ways in which access to widearea network 100 may be obtained for a participant device.

Video communication data for video communications may itself beexchanged utilizing various transportation formats or protocols suitableto or capable of being communicated via wide area network 100, includingbut not limited to SIP (Session Initiation Protocol), STUN (SessionTraversal Utilities for NAT), TURN (Traversal Using Relay NAT), ICE(Interactive Connectivity Establishment), RTP (Real-time TransportProtocol), SRTP (Secure RTP). Video communication modules may beconfigured to format, process, transmit, send, or receive according tothe various transportation protocols, as well as encrypt, compress,encode, decrypt, decompress, and decode video communication data (e.g.,audio and/or video data).

FIG. 2A is a block diagram illustrating dynamic display of videocommunication data at a participant device in a video communication,according to some embodiments. Participant device 200 may be any one ofparticipant devices 102, 112, 122, 132, 142, 152, or 162, of portablemulti-function device 3000 described below with regard to FIGS. 22-24 orcomputing system 4000 described below with regard to FIG. 25.

Participant device 200 may implement multiple layers of functionalityfor performing various tasks or applications, such as videocommunication. Application layer 210 may provide for the implementationof applications, such as video communication application 214 (similar tovideo communication modules discussed above with regard to FIG. 1) aswell as many other application(s) 212. Applications implemented inapplication layer 210 that are currently executing on participant device200 may access data, features, services, controls, or otherfunctionalities of participant device 200 via one or more differentobjects or programmatic interfaces (APIs) for the respective data,features, services, controls, or other functionalities of participantdevice 200 implemented in middleware/operating system layer(s) 220.Participant device 200 may implement various middleware layer(s) 220and/or an operating system layer 220 in order to provide services. Anoperating system may, for instance, provide access to and/or managementof hardware devices implemented in hardware layer 230, such as displaydevice 232, or other devices, components, or subsystems, or circuits.Participant device 200 may include hardware layer 230 which includesvarious hardware components, such as electronic display 232, otherinput/output devices (e.g., microphones, keyboards, touch-sensitiveareas—which may also be electronic display 232, speakers) or otherhardware components that implement participant device 200, processors,graphics accelerators, or other components (such as those discussedbelow with regard to portable multi-function device 3000 or computingsystem 4000).

As illustrated in FIG. 2A, video communication application 214 may beimplemented in application layer 210. Video communication application214, such as those video communication modules discussed above withregard to FIG. 1 or in more detail below with regard to FIG. 3, mayperform video communications with other participants over networks, asdiscussed above with regard to FIGS. 1A and 1B. Video communication data202 may be received at video communication application 214 as part of acurrent (or initiated) video communication with one or more otherparticipant devices. Video communication application 214 may beconfigured to process video communication data 202 (e.g., decrypt,un-compress, decode) or otherwise prepare video communication data fordisplay. Video communication application 214 may then as part ofperforming the current video communication make one or more calls orrequests to audio/video management module 222 in order to ultimatelydisplay the video communication data on display 232. In variousembodiments, video communication data may include video communicationdata from multiple different participants participating as part of thesame video communication (e.g., a multi-way video communication).

In various embodiments, video communication 214 may operate in differentdisplay modes. Full-screen display mode, for instance, may allow videocommunication application 214 control of most or all of the display 232for implementing the video communication including other video/imagedata for user interface elements or graphics (e.g., windows, toolbars,or buttons) that format or frame the video communication data displayed.In at least some embodiments, integrated display mode may be employedwhen another application 212 or components (e.g., an operating systeminterface) is displaying image data, video data, or any other form ofgraphical display on display 232. Audio/video management component 222may provide software hooks, display attributes, or other techniques todirect the display of video communication data in either the full-screenmode or the integration display mode. For example, if videocommunication application 214 is operating in full-screen mode, thenaudio/video management component may determine a display location withindisplay 232 for the video communication data 202 that is specified orrequested by video communication application 214, which may consume mostof display 232. FIG. 2C, for instance, illustrates a participantdisplaying video communication data in a full-screen mode for a videocommunication, according to some embodiments.

In some embodiments, video application module 214 may be operating inintegration display mode. For instance, other application 212 may becurrently executing. As part of the execution of other application 212,application image data 204 (or video or other graphics data) may beobtained, received, or generated and displayed on display 232, asindicated by the arrow from other applications 212 through audio/videomanagement module 222 to display 232. Video communication data 202 for avideo communication initiated prior to or during the execution of otherapplication 212 may also still be received. Audio/video managementmodule 222 may be configured to identify a display location withindisplay 232 for video communication data 202. This identification may bemade dynamically as display attributes for the video communication datamay be configured at runtime by other application 212. For example,other application 212 may define software hooks, or event detectionmechanisms that configure display attributes if video communication datais received for display. In this way, video communication data 202 maybe displayed in a location that is optimized to allow app image data 204to be displayed as intended, even though a current video communicationmay be ongoing, as indicated by the arrow within audio/video managementmodule 222. In at least some embodiments, video communication data maybe blended or integrated directly with app image data 204 or other videoor graphic data encoded as a single stream of data for display, asdiscussed below with regard to FIG. 5. Audio/video management module 222may coordinate or manage software or hardware resources (e.g., videoencoders) which may be utilized for app image data 204 and videocommunication data 202.

FIG. 2B illustrates a participant device displaying video communicationdata in an integrated display mode for a video communication, accordingto some embodiments. Video communication data 202 is integrated with appimage data 204 according to the display attributes configured by otherapplication 212, placing the video communication data 202 in a locationwithin participant device display 232 that does not hinder display ofapp image data 204 (e.g., a blank or dead space, or within image datathat may be easily adapted to work around video communication data 202).Please note that although the above example integration display mode isgiven with regard to an application in application layer 210, it may bealso that a component of middleware/operating system layer(s) 220 (e.g.,an operating system user interface) may also configured displayattributes for video communication data so as to display videocommunication data 202 in a way that accommodates image or othergraphics data displayed for the component of middleware/operating systemlayer(s) 220.

In various embodiments, audio/video management module 222 may beconfigured to determine the display mode of a current videocommunication, whether full-screen or integration display mode. However,in some embodiments, video module application 214 may determine ormaintain the state of the display mode and provide an indication of, orformat requests to, audio/video management component 222 so thataudio/video management module 222 handles video communication data 202according to the current display mode. Input may be received, in someembodiments, to switch between full-screen display mode and integrateddisplay mode.

Video application 214 may also be configured to provide to audio/videomanagement 222 for display user interface elements. These user interfaceelements may also be displayed according to the determined location(e.g., within the determined location as specified by video applicationmodule 214) or various other locations in display 232 (e.g. as may beindicated by other display attributes configured at runtime by otherapplication 212). Video communication application 214 may handle inputreceived via these user interface elements (e.g., pause, resume, end).

Video communication applications may be implemented in many differentways in order to facilitate video communications over networks. FIG. 3is a block diagram of a video communication module, according to someembodiments, that may implement video communications. Please note thatthe various components illustrated in FIG. 3 may be implemented inhardware, software, or any combination thereof. Where single componentsare illustrated it may be understand that multiple components mayalternatively be implemented to provide the same functionality. Thus,the following discussion regarding FIG. 3 is not intended to be limitingas to the implementation of a video communication module. Videocommunication module 300 may be implemented on a variety of differenttypes of participant devices, such as mobile computing devices, mobilephone, laptops, or other portable multi-function devices (such as videocommunication module 3039 implemented as part of portable multi-functiondevice 3000 in FIG. 22) or computing devices (such as computing system4000 described below with regard to FIG. 25).

In various embodiments, video communication module 300 may implement auser interface 310 to receive, process, parse, or handle input for thevideo communication module 300 from a user. For example, in someembodiments, user interface 310 may provide multiple user interfaceelements that are selectable via an I/O device. If, for instance, theparticipant device implements a touch-sensitive display, then the userinterface elements may be selected by touch, gesturing, or otherwiseconveying selection of a particular user interface element. In someembodiments with a touch-sensitive display, user interface elements neednot be displayed, but instead the touch gestures may themselvesinterpreted or parsed to indicate or convey the election of certainactions via the user interface. User interface 310 may implement manydifferent elements for video communication module 300, including, butnot limited to, initiating a video communication (either in a 2-way orin a multi-way mode with other participant devices), ending a videocommunication, pausing a video communication, resuming a videocommunication, switching between display modes, formatting display ofvisual communication data, selecting input devices (e.g., cameras),muting audio, recording audio/video/still images of video communicationdata, or any other component that provides input or direction to videocommunication module 300 from a user. Other input devices, such ascomputer mice, keyboards, voice commands or recognition, physicalmovement of a participant device (turning, tipping, or flipping thedevice) may also be input, translated or detected via user interfacecomponent 310, in some embodiments.

Video communication module 300 may implement connection manager 320, invarious embodiments. Connection manager 320 may be configured toestablish connections for video communications with video communicationparticipant(s) 304. For instance, connection manager 320 may accessrecipient contact data 250 for identification information to establish aconnection one or more participant devices (as may be requested by auser). Various different protocols or formats may be implemented byconnection manager 320 to establish video communication connections,such as those discussed above with regard to FIGS. 1A and 1B. In someembodiments, connection manager 320 may be configured to maintaininstant or pre-established connections so that some video communicationsmay appear to be initiated and/or displayed in real-time (as discussedbelow with regard to FIGS. 18-21). Connection manager 320 may also beconfigured to handle dynamic transitions from video messages received ata participant device into a live video communication, as well astransitions back to an offline video messaging format, in variousembodiments (as discussed below with regard to FIGS. 14-17).

In at least some embodiments, video communication module 300 may haveaccess to one or more persistent storage devices or data stores thatmaintain recipient contact data 350 and stored video communication data360. In some embodiments, recipient contact data 350 may be identities,address, or other network/transmission information to establish a videocommunication connection and transmit/receive video communication datavia the video communication connection with a particular videocommunication participant. In some embodiments, stored videocommunication data 360 may be stored versions of currently ongoing orprior video communications (as well as other data related to them, suchas still images or audio recordings). Stored video communication data360 may be accessed for playback or display, either by videocommunication module 300 or another application, component of aparticipant device. In some embodiments, stored video communication data360 may not be stored locally with respect to the participant deviceimplementing video communication module 300, but may instead be storedin a remote storage location (e.g., a media server) accessible to videocommunication module 300.

As illustrated in FIG. 3, video communication module 300 may receivevideo communication data (video and/or audio data) from videocommunication capture device(s) 302). For example, an on-board videocamera and/or on-board microphone (e.g., optical sensors 3064 and/oraudio circuitry 3030 in FIG. 22), externally coupled video camera and/orexternally coupled microphone may be used to capture video and/or audiodata for transmission to video communication participant(s) 304. Videocommunication module 300 may implement transmission/reception manger 330to handle outgoing and incoming video communication data for a videocommunication, in some embodiments. For instance, transmission/receptionmanager may encode, encrypt, compress and/or otherwise format capturedvideo communication data for transmission to video communicationparticipant(s) 304 and direct the sending the video communication dataaccording to various communication protocols to video communicationparticipant(s) 304, such as discussed above with regard to FIGS. 1A and1B. For incoming video communication data for a video communication,transmission/reception manager 330 may be configured to decode, decrypt,decompress, or otherwise reformat data for display. Display manager 340may direct the display of received video communication data according toeither a full-screen display mode and/or an integration display mode, insome embodiments. Display manager 340 may also provide the videocommunication data to an audio/video management component (e.g.,audio/video management module 222 in FIG. 2) or another application forfurther processing prior to display.

In various embodiments, communication module 300 may implement variousones of the methods, techniques, and/or components discussed below withregard to FIGS. 4-21. However, some embodiments, may implement onlyparticular ones, such as only techniques of FIG. 4, FIG. 12, FIG. 14, orFIG. 17. Thus, FIG. 3 is not limited to any particular combination orarrangement of components or implementation of techniques describedbelow.

As noted previously, dynamic display of video communication data may beimplemented as part of various different components or devices of aparticipant device. An application that is itself executing on theparticipating device may perform the following techniques in order toincorporate video communication data into graphics displayed for theapplication on an electronic screen. Similarly, a video applicationmodule itself may be configured to implement the techniques such thatthe display location for video communication data is dynamicallydetermined even though another application may be controlling anelectronic display. Alternatively, as illustrated above in FIG. 2A, amiddleware/operating system component may implement the followingtechniques. FIG. 4 is thus a high-level flowchart illustrating methodsand techniques for dynamic integration of video communication data,according to some embodiments, which may be implemented by varioussystem components or devices.

As indicated at 410, video communication data may be received fordisplay as part of a current video communication. The videocommunication data may include video and/or audio data to be displayedor presented. In various embodiments, if a video communicationapplication is operating in full-screen mode, then the videocommunication data may be displayed in an electronic display accordingto a full-screen mode, as indicated at 420. The full-screen display modemay, in some embodiments be controlled by the video communicationapplication. However, other applications may also trigger or initiatefull-screen display mode, in some embodiments. In some embodiments,video communications may be displayed in full-screen display mode bydefault, unless otherwise specified by another application, operatingsystem/middle ware component, or the video communication application. Ifno other application is controlling the electronic display, as indicatedby the negative exit from 430, then the video communication data maycontinue to be displayed in full-screen display mode (or as otherwisespecified by the video communication application.

If another application is launched, initiated, resumed, woken up, orotherwise begun such that the other application is controlling theelectronic display in order to display image data, as indicated by thepositive exit from 430, then a display location for the videocommunication data may be dynamically determined according to displayattributes configured at runtime as part of executing the otherapplication, in various embodiments, as indicated at 440. Otherapplications launched may be any application that utilizes or controlsthe electronic display, from games and other media, to othercommunication applications (including other video communicationapplications or screen sharing applications), as well as various tools,tasks, or other processes that an application may be configured toperform. Software hooks, event listeners, middleware or operating systemprovided API calls or other devices may allow the other application todefine where, how, and/or when the video communication data isdisplayed, defining and redefining the display attributes as theapplication sees fit. Dynamically determining a display location may, inmany embodiments, allow the other application to blend or integrate thevideo communication data into data that is received, generated, orotherwise obtained via the current video application. For example, FIG.5 illustrates a participant device 200 that displays integrated videocommunication data 500. In FIG. 5, a gaming application may receivevideo communication data (e.g., the image of a user of the otherparticipant device) which may blend the image data of the user into aparticular scene of the game (e.g., as an opponent in a dominoes game)at particular location within the electronic display. Stated moregenerally, the other application may be able to control, transform,modify, or otherwise adapt the video communication data to be rendereddifferently on the electronic display in a way that is different thanthe video communication data would be displayed as originally received.Thus, the video communication data may be dependent on the otherapplication such that a single video stream may be generated for displaythat includes the video communication data, instead of just displayingan independent stream of video data.

As indicated at 450, the video communication data may then be displayedat the determined display location within the electronic display. If nochange in the display mode is made (e.g., a switch to full-screendisplay mode), as indicated by the negative exit from 460, then thedetermination of a display location for video communication data, asindicated at 440, and the display of the video communication data 450may be repeated. In this way, the other application may change thedisplay location of the video communication data over time. FIGS. 6Athrough 6C illustrate different display locations for videocommunication data that may be determined over time. For example,display location 610 for the video communication data in FIG. 6A may beinitially determined when the other application first begins executing.If however, the other application changes (e.g., in response to userinput) the display of data on the electronic display, then the displaylocation for the video communication data may also change. For instance,consider an example of the other application that is shared book readingexperience via the electronic display. In between FIGS. 6A and 6B, avirtual page in the shared book application is “turned,” so the locationwherein the video communication data may be best displayed has changedto display location 620. Similarly, display location may change againbetween FIGS. 6B and 6C, changing from display location 620 to displaylocation 630.

As indicated by the positive exit from 460, a change to full-screendisplay mode may change the control of displaying video communicationdata back to the video communication application. Please note, that insome embodiments, a video communication may be started while the videocommunication application is considered to be in integration displaymode. For instance, the other application may already be running anddisplaying graphics on the electronic display. Thus, after receiving thevideo communication data, as indicated at element 410, a displaylocation for the received video communication data may then bedynamically determined, as indicated at 440 (without first displayingthe video communication in full-screen display mode). FIG. 7, discussedbelow provides a similar example of initiating video communications inintegration display mode, and although it describes a multi-way videocommunication the same techniques may be applied for a videocommunication between two participants, in at least some embodiments.

FIG. 7 is a high-level flowchart illustrating methods and techniques fordynamically displaying video communication data for multipleparticipants in a video communication, according to some embodiments. Asindicated at 710, image data (or other video or graphics) may bedisplayed for an application within an electronic display. While theapplication is running, a video communication may be initiated withmultiple users, as indicated at 720. For example, an interface elementor other component may be selected to start the video communication.Alternatively, an incoming connection for the video communication withmultiple participants may be received, in some embodiments.

As indicated at 730, video communication data for the videocommunication data may be received that includes video communicationdata for each of the multiple participants, in various embodiments. Forinstance, separate data streams transmitted from each of theparticipants may be received. One or more display locations may bedynamically determined for the video communication data within theelectronic display according to display attributes configured at runtimeas part of executing the application, as indicated at 740. Once thedisplay locations are determined for the multiple participants, therespective video communication data corresponding to each participantmay be displayed in the determined display location, as indicated at750. Please note, that video communication data for different respectiveparticipants may arrive at different times, and thus the videocommunication data for each participant may be displayed independentlyof the other participants. For example, video communication data forparticipant C may be displayed prior to video communication data forother participants. In some embodiments, as participants are added to orleave the video communication, the display of the respective videocommunication data for the additional participant may be added(according to a dynamically determined display location) or removed.

For example, FIG. 8 illustrates dynamically displayed videocommunication data in an electronic display of a participant device formultiple participants in a video communication, according to someembodiments. Different respective display locations in the electronicdisplay of participant device 200 may be determined for multipleparticipants. For video communication data for participant A, displaylocation 810 is determined. For video communication data for participantB, display location 820 is determined. For video communication data forparticipant C, display location 830 is determined. In some embodiments,the other application may be another video communication or transmissionapplication, such as a screen sharing application. Other videocommunication data 840 may, for instance, be information shared amongthe respective electronic displays via the screen sharing applicationfor participants in the video communication.

Input for a video communication in integrated display mode may still behandled, even though another application is executing and controllingthe electronic display. The video communication application may, in someembodiments, handle the received input, and imitate appropriateresponses. FIG. 9 is a high-level flowchart illustrating methods andtechniques for handling input for video communication data that isdynamically displayed, according to some embodiments. As indicated at910, video communication data may be displayed according to dynamicallydetermined display locations within the electronic display. Thedetermination of display locations may be performed, as discussed above,with regard to display attributes configured as part of the execution ofan application different than a video communication application. Asindicated at 920, input for the video communication may be detected. Forexample, the electronic display may be touch sensitive and one or moretouch gestures or input may be detected via the electronic display, insome embodiments. Other input devices, such as computer mice,trackballs, keyboards, or voice commands, may also communicate input forthe video communication.

If input is detected, as indicated by the positive exit from 920, theinput may be analyzed or parsed to determine a response, in variousembodiments, as indicated at 930. For example, one or more userinterface elements may be displayed, and selection of a particular userinterface element corresponding to a particular response may bedetermined. These user interface elements may, in some embodiments bedisplayed dynamically as well, under the control of the otherapplication. In another example, specific gestures for a touch-sensitiveinput device, such as a touch-sensitive electronic display, may beinterpreted and correspond to particular actions.

One response that may be determined is ending the video communication,as indicated at 940. For example, a user interface element for endingthe video communication may be selected, and in response the videocommunication connection may be terminated. In another example, aparticular gesture, such as a particular touch swipe in a particulardirection may also be interpreted as a request to end the videocommunication, such as touch input 1012 illustrated in FIG. 10A.

A response that may be determined is the display of user interfaceelements for the video communication, as indicated at 950. For example,FIG. 10A illustrates a participant device 200 displaying videocommunication data for a current video communication at display location1010. Touch input 1012 may be detected within display location 1010. Asillustrated in FIG. 10B, additional user interface elements 1020 may bedisplayed within display location 1020, which may allow for furthercontrol of the video communication (e.g., pause, resume, end, or switchto full-screen display mode). Further touch input with respect to theseuser interface elements 1020 may then be detected, in some embodiments.

Another response that may be determined is the resumption of the videocommunication, as indicated at 960. For example, the touch input 1012 inFIG. 10A may be particular gesture (e.g., a single tap) to resume thevideo communication from a paused state, in some embodiments. Similarly,another response that may be determined is a pause of the videocommunication, as indicated at 970. A pause user interface element maybe selected, for instance, or a particular action or touch gesture takenwith respect to the participant device (e.g., turning the participantdevice another direction).

A response may be to switch from an integration display mode to afull-screen display mode, as indicated at 980. For example, the touchinput 1012 in FIG. 10A may be particular gesture (e.g., a double tap ora directional swipe) to switch to full-screen display mode, in someembodiments. As illustrated in FIG. 10C, participant device 200illustrates the video communication in full-screen display mode 1030after the input 1020 illustrated in FIG. 10A.

As noted above, dynamic pause and resume may be implemented for a videocommunication, according to some embodiments. Mobile computing devices,such as mobile telephones, laptops, personal digital assistants (PDAs),portable multi-function computing devices (e.g., portable multi-functioncomputing device 3000 in FIGS. 22-24), or computing systems (e.g.,computing system 4000) may have limited connectivity or access tonetworks over which video communications may be transmitted. Videocommunications, like in-person conversations, may experiencefluctuations in the active exchange of information. However, videocommunications may continue to transmit video communication data eventhough little or no active exchange of information is occurring. Dynamicpause and resume may allow for the transmission of video communicationdata to correspond to the exchange of information between participantsin a video communication. The transmission of data for the videocommunication from a participant device may be evaluated to dynamicallydetermine whether the transmission of video communication data (or atleast some of the video communication data) may be paused or resumed.

FIG. 11 is a block diagram illustrating an exampletransmission/reception manager module for a video communication moduleimplementing dynamic pause and resume for transmission of videocommunication data, according to some embodiments. Video communicationmodule 300 may implement transmission/reception manager 330, asdiscussed above with regard to FIG. 3. Captured video communication data1102 may be received at transmission/reception manger 330 to betransmitted 1106 to participant devices as part of the videocommunication. The stream of captured video communication data may beevaluated. For example, transmission/reception manager 330 may implementactive exchange dynamic evaluator 1110 to dynamically evaluate thestream of video communication data to detect pause or resume events.Various different video analysis techniques (e.g., facial recognition),audio analysis techniques, other sensor information, as well as userprovided information (e.g., interaction with applications at theparticipant device) may be implemented by active exchange dynamicevaluator 1110 to evaluate the stream of video communication data. FIG.13, discussed below, provides further examples of techniques that may beimplemented to detect pause or resume events. Once detected, the pauseor resume events may be conveyed to transmission state manager 1140.

Transmission/reception manager 330 may implement transmission statemanager 1140 to maintain the current transmission state of a videocommunication. For example, the transmission state manager 1140 mayindicate whether the video communication is paused, as well as whatvideo communication data may be allowed to be transmitted (if any) whenthe video communication is paused. If, for instance, video data isfiltered out, while audio data for the video communication is stilltransmitted, then transmission state manager 1140 may indicate that thevideo data is filtered out. In various embodiments, indications of userselection(s) 1104 to manually pause and resume the video communication(as opposed to the pause and resume events detected by active exchangedynamic evaluator) may also be received and update the indicated stateof the transmission at transmission state manager 1140.

In some embodiments, transmission/reception manager 330 may implementvideo communication data stream filter 1120. Dependent on the currenttransmission state, video communication data stream filter 1120 maypause or filter some of the transmission data, such as video data, audiodata, or both video and audio data, in some embodiments. Filter 1120 mayalso filter or select different amounts of the stream of videocommunication data to send (e.g., audio only or lower amount of videodata) even when not in a pause state (as some recipient devices may onlyreceive portions of the stream of video communication data).Transmission/reception manager 330 may implement video communicationdata stream transport formatter 1130. Filtered data (or unfiltered dataif full transmission is resumed) may then be formatted at videocommunication data stream transport formatter 1130 to be sent accordingto the established connection for the video communication. Varioustechniques for encoding, encrypting, compressing, and/or formatting thecommunication data as well as constructing messages, packets, or othertransportation containers commensurate with communication protocolformats, such as real-time protocol (RTP) format, may be implemented toprepare the video communication data for transmission. The videocommunication data may then be transmitted 1106.

FIG. 12 is a high-level flowchart illustrating methods and techniquesfor dynamically detecting pause or resume events for a videocommunication, according to some embodiments. As indicated at 1210, astream of video communication data may be captured for transmitting to aparticipant of a current video communication, in some embodiments. Thecapture of video communication data, as discussed above with regard toFIG. 3, may be performed by various optical and/or auditory sensors(e.g., cameras and/or microphones). The video communication may, invarious embodiments, be peer-managed, as discussed above with regard toFIG. 1. As indicated at 1222, the stream of video communication data maybe dynamically evaluated, in some embodiments. FIG. 13, discussed below,provides discussion of various evaluation techniques. For instance thevideo data and/or audio data within the video communication data may beanalyzed, as well as various other input from other sensors orcomponents. Based on the evaluation, a pause event may be detected, asindicated at 1230.

If a pause event is detected, as indicated by the positive exit of 1230,the video communication may be paused (for which at least some of thestream of capture video communication data is then not transmitted to aparticipant(s) in the video communication). Note, that in variousembodiments, not all recipient devices that are participants in a videocommunication may receive the same amount of data. For instance, someparticipant devices may be more resource challenged (e.g., preventingthe device from processing the entire stream of video communication dataefficiently), than others. In another example, different users may haveaccess or privilege to different amounts of the video communication datastream from the sending device, or a user of the sending device may wishto overlay graphic effects or other information to a particularrecipient in multi-way video communication. Correspondingly, pausing thetransmission of video communication data may reduce or halt transmissionaccording to the respective amounts of the stream of video communicationdata that were transmitted originally. If, for instance, one recipientonly receives audio data as part of the video communication and anotherrecipient receives audio and video data, then in the event of a pause atthe sending device, the first recipient may receive none of the streamof video communication data while the second recipient may receive onlyaudio data. Alternatively, pausing may reduce all transmission of thestream of video communication data to a particular amount (or none atall) in some embodiments.

Alternatively, if the current transmission state for the videocommunication is paused, then a resume event may be detected, asindicated by the positive exit from 1250. Transmission of at least aportion stream of video communication data may be resumed or transmittedagain, as indicated at 1260. For example, if in a pause state no data(or only audio data) is transmitted, upon detection of a resume eventboth audio and video data may be transmitted to the recipientparticipant. Again, as noted above, different recipients in a multi-wayvideo communication may receive differing amounts of the stream of videocommunication data. When resuming transmission of at least a portion ofthe stream of video communication data, the amount of videocommunication data that was originally sent to a participant recipientmay be resumed (e.g., if only audio was sent, and during a pause no datawas sent, then at resume only audio may be sent). Circumstances forparticipants in a multi-way video communication may change during apause state for a sending device. For instance, a recipient device thatwas resource challenged may no longer be resource challenged (andalternatively a recipient device may become resource challenged).Whatever the cause, upon resume, in some embodiments different amountsof the stream of video communication data than were originally sentprior to the pause may be sent to recipient devices.

In some embodiments, manual input for pause and resume events may bereceived. As indicated at 1220, indication of a user selection of atransmissions state for the video communication may be received. Forinstance, a user interface element may be selected, touch input, audioinput, or some other form of input may be received and interpreted tocorrespond to pausing or resuming the video communication.

Dynamic detection of pause and resume events may be beneficial whenoperating multi-way video communications and/or displaying videocommunication data in integrated display mode. When, for instance,multiple different participants are transmitting and receiving videocommunication data for a video communication, dynamic pausing of thevideo communication data may lessen the computational burden on thetransmitting and receiving participant devices. Similarly, the automatedor dynamic nature of detecting pause and resume events may prevent auser from manually pausing and resuming transmission when operatingother applications at the same time as the video communication (as maybe the case in integrated display mode).

FIG. 13 is a high-level flowchart illustrating methods and techniquesfor determining an active exchange indicator of a video communication,according to some embodiments. As discussed above, video communicationdata may be captured 1300 for transmission. Video data may be analyzed1310, in various embodiments. For example, facial recognition techniquesmay be performed to identify whether or not a person is in view of theparticipant device. Other forms of video analysis may be employed. Forinstance, light levels may be determined, or an amount of motion may bedetected. As indicated at 1312 audio data may be analyzed, in someembodiments. Speech analysis may be employed to detect an amount ofconversation or speech. Background noise or other audio signals may alsobe evaluated or recognized (and ignored). For example, it may bedetermined that ongoing speech is not directed to other participants ofthe video communication. Other sensor input 1314 may also be analyzed.For instance, orientation sensors, such as included in portablemulti-function device 3000 described below with regard to FIG. 22, maybe used to detect motion of the participant device. In some embodiments,other sensor input 1314 may indicate whether another participant devicelinked to a same user account as the device transmitting data is alsotransmitting video communication data and/or whether the otherparticipant device linked to the same user account is in proximity tothe device. If it may be determined based on analyzing audio or videodata received from the other device, that a user is instead using theother device to participate in the video communication, then it mayindicate whether or not to pause video communication at the device. Forexample, if a mobile phone and a tablet computer are both linked to thesame user account, and both devices are participating in a videocommunication, then a pause event may be detected on one of the devices(e.g., the tablet computer) based on the active exchange of informationdetermined at the mobile phone and/or based on the proximity of themobile phone to the tablet computer.

As indicated at 1320, in some embodiments a determination of an activeexchange indicator may be made. For example, a weighting scheme forvarious different determinations made at elements 1310, 1312, and 1314may be implemented, in some embodiments. A face that is recognizedwithin the view of the participant device may weighted as more likely toindicate active exchange of information if combined with particularamounts of detected speech. Other user-provided information may beconsidered when determining an active exchange indicator. For instance,if no face is recognized, but an it is also known that an outward facingcamera is selected (one which would not be pointed at a user but pointedso as to show a recipient something that is not the user), then the lackof facial recognition may be negated. Thus, in various embodiments thesource or specific devices capturing the stream of video communicationdata may be considered when evaluating the data stream to detect pauseand resume events.

Once the active exchange indicator is determined, the active exchangeindicator may be compared with a particular exchange threshold. Whethera resume event or a pause event it is triggered may depend on thecurrent transmission state of the video communication. As indicated bythe negative exit at 1330, if the current transmission state is notpaused, then the active exchange indicator may be compared with anexchange threshold. If the active exchange indicator is below theexchange threshold, as indicated by the positive exit from 1340, then apause event may be triggered, as indicated 1350. If not, as indicated bythe negative exit from 1340, then more data may be captured 1300 and theanalysis performed again. As indicated by the positive exit from 1330, acomparison between the active exchange indicator and whether it is abovethe exchange threshold, as indicated at 1360, may be made. If the activeexchange indicator is above the exchange threshold, as indicated by thepositive exit from 1360, then a resume event may be triggered. If not,then as indicated by the negative exit from 1360, then more data may becaptured 1300 and the analysis performed again.

Initiating a video communication typically involves both an initiatingdevice and a recipient device. A video connection may be established ifthe recipient accepts video communication. As video communicationbecomes more common, more entities unknown to a recipient may request avideo communication, or more video communications even between knownparties may be sought. However, video communications may not alwaysdesirable, unless it can be ascertained as to the subject of thecommunication. Instead of initiating a video communication, a videomessage may be sent to a recipient, which can be replayed to ascertainthe nature of the video communication. If it is desirable, the recipientdevice can initiate a dynamic transition of the video message into avideo communication. FIG. 14 is a high-level flowchart illustratingmethods and techniques for a dynamic transition from a video message toa video communication, according to some embodiments.

As indicated at 1410, video communication data may be received via avideo messaging protocol from a remote mobile computing device that ispart of a video message from the remote mobile computing device, invarious embodiments. For example, video communication data may be sentand/received via a multi-media messaging (MMS) protocol (which may bepartially carried over a different network, such as a wireless telephonenetwork, than the wide area network for which video communications maybe transmitted). Alternatively, video communication data of a videomessage may be sent via an instant messaging or chat protocol.

As the video communication data of the video message is received, thevideo message may be displayed on an electronic display at the recipientdevice, as indicated at 1420. For example, the video message may bedisplayed at the recipient device in near real-time, which may “pop up”or automatically be displayed—even if another application is currentlydisplaying or controlling the electronic display or if the electronicdisplay is “asleep” or not active, similar to the integration displaymode discussed above. The video message may also be recorded forsubsequent display at the recipient device, as indicated at 1430. Forinstance, the video communication data for the video message may bestored at local storage on the recipient storage device. In variousembodiments, the video communication data may be recorded in a lesserquality or different version (such as may be achieved by a differentvideo encoding like open format encoding) than the quality or version ofthe data as it is received.

In at least some embodiments, an indication to initiate a videocommunication with the remote mobile computing device may be received,as indicated by the positive exit from 1440. In various embodiments, auser selection of a user interface element or some other form of inputreceived at the recipient device may indicate that the videocommunication is to be initiated. For instance, a touch gesture, such astapping the displayed video message on a touch-sensitive electronicdisplay, may indicate the desire to initiate a video communication.

In response to receiving the indication to initiate the videocommunication, a video communication connection may be established withthe remote mobile computing device, as indicate at 1450, in variousembodiments. For instance, various handshake messages or othercredential exchanges may be performed in order to establish theconnection. Subsequently received video communication data may then bereceived via the established video communication connection which may bedifferent than the video messaging protocol. For example, the videocommunication may be performed via a peer-to-peer and/or peer-managedimplementation of a video transport protocol, such as real-timetransport protocol (RTP). Additionally, video communication data may becaptured and transmitted from the recipient device to the remote mobilecomputing device via the established connection (as may be performed aspart of a two or multi-way video communication). In at least someembodiments, the video communication data received one the videocommunication connection is established may continue to be recorded andstored along with the video communication data received via the videomessaging protocol for subsequent playback.

In at least some embodiments, though the video communication connectionis established and may allow real-time communication, captured videocommunication data may still be exchanged as a message via theestablished video communication channel. For instance, a user mayindicate in the selection to establish a video communication connectionthat the conversation may still occur “offline” (i.e., by sending andreceiving video messages). Exchanged video messages may be sent via theestablished connection (and according to is various protocolrequirements). Moreover, indications may be received to switch betweenan offline conversation and a real-time conversation with the videocommunication connection already established. In some embodiments,however, the way in which “offline” video messages may be transmittedmay be determined based on a cost or responsiveness determination. Forinstance, if resources used to establish the video communicationconnection may be used for other purposes, then the video communicationmay be relinquished and video messages exchanged via the video messagingprotocol.

In at least some embodiments, no indication may be received to initiatethe video communication, or a negative indication to not initiate thevideo communication may be received, as indicated by the negative exitfrom 1440. This indication may, in some embodiments indicate a desire tocease displaying the video communication data as it is received, asindicated at 1460. For example, a touch gesture, such as swiping awaythe displayed video message on a touch-sensitive electronic display, mayindicate the desire to cease display of the video communication data(and not initiate a video communication). Recording of the videomessage, however, may continue. In some embodiments, the stored videocommunication data of the recording may be subsequently displayed, inresponse to a playback request. Although not illustrated, in someembodiments, an indication may be subsequently received to establish avideo communication (after initially receiving an indication not todisplay the video communication). For instance, a voice command may bedetected to instruct the display and establishment of a videocommunication with the sending device who sent (or is still sending) thevideo message.

The transition between a video message communication, which isunilateral or one-way, into a two-way or multi-way video communicationmay also be employed to handle network changes that may make the videocommunication unsustainable (e.g., due to poor network connectionquality). FIG. 15 is a high-level flowchart illustrating methods andtechniques for monitoring the quality of video communication connectionfor dynamic transitions between video communications and video messages,according to some embodiments. As indicated at 1510, video communicationdata may be captured for transmission to a remote mobile computingdevice as part of a video communication. The captured videocommunication data may be sent via an established connection to one ormore remote mobile computing devices. As indicated at 1520, the videocommunication may be monitored to determine a connection qualityindicator. For example, various signal strength metrics (for wirelessconnections) or network performance metrics (e.g., available bandwidthor connection speed) may be used to determine the connection qualityindicator.

If the video communication maintains a connection quality indicator ator above a certain tolerance threshold, then the video communication maycontinue, as indicated by the negative exit from 1530. However, invarious embodiments if the connection quality indicator falls below thetolerance threshold, then a transition to video message protocol may bemade. For example, as indicated by the positive exit from 1530, anindication of transition to video message protocol may be provided, asindicate at 1540. This indication may, in various embodiments, be analert, notification, message or information display at the transmittingparticipant device. In some embodiments, a notification or otherindication may also be provided to other participant(s) in the videocommunication that the particular participant device has switched to avideo message protocol. As indicated at 1550, the captured videocommunication data may be recorded to send as a video message via thevideo messaging protocol to the remote mobile computing device. Althoughnot illustrated, a reverse transition may be made if, for example, theconnection quality indicator meets or rises above the tolerancethreshold, then captured video communication data may be sent again viathe established video communication connection, in at least someembodiments. For example, in some embodiments it may be determined thatthe connection quality indicator equals or is above the connectiontolerance threshold, and in response the transmission of captured videocommunication data may be resumed via the video communicationconnection.

Although previously discussed in the context of receiving a videomessage to initiate a two-way video communication, the same techniquesmay be applied for larger groups of participant devices. For example,multiple participant devices communicating via a group message thread orvideo chat thread may implement the above techniques. FIGS. 16A and 16Billustrate a dynamic transition for multiple participants in a videomessage to a video communication, according to some embodiments. FIG.16A illustrates a group message thread 1600. Different participants inthe group message thread 1600 may send/post different video messages,such as participant A video message 1610, participant B video message1620, and participant C video message 1630. The video messages may bedisplayed as they are received. Moreover, as discussed above with regardto FIG. 14, an indication to initiate a video communication may bereceived, such as touch input on one or more of the video messages 1610,1620, and/or 1630. FIG. 16B illustrates a dynamic transition to a videocommunication with participants of the group message thread 1600. Videocommunication data for participant A 1612, video communication data forparticipant B 1622, and video communication data for participant C 1632may be displayed as part of a multi-way video communication. Pleasenote, that although all participants are illustrated as participating inthe multi-way communication, not all group message thread participantsneed or accept participation in a live video communication. For thoseparticipants that do not participate, video messages containing thecommunication data may be transmitted or posted to the group messagethread, in some embodiments. In this way, some participants of a groupmessage thread may passively participate in the group communication.Participants that later join either the group chat, message thread, ormulti-way video communication may also receive (and possibly display)prior video communications or messages for the group communication(e.g., replaying the history of the conversation), in some embodiments.

As noted above with regard to FIG. 14, in various embodiments videocommunication data may be recorded, whether transmitted via the videomessaging protocol or the video communication connection. In someembodiments, the communication data may be stored locally at eachparticipant's device. However, large amounts of video communication datamay exceed local storage resources. The video communication data maythus be stored in a different (e.g., lower resolution) format. In someembodiments, video communication data may be stored at a remote storagedevice or data store, such as discussed above with regard to storedvideo communication data 360 in FIG. 3.

Various privacy safeguards and protocols may be implemented, however,for any video communication data that is remotely stored. For example,consent to record video communications remotely (or locally on otherparticipant devices) may have to be explicitly obtained (e.g., aconfirmation dialogue interface element may have to be selected orapproved). The video communication data may be encrypted such that onlyparticipants in the video communication may be able to access the storedvideo communication data. In some embodiments, strict data retentionpolicies may be implemented such as to remove the stored communicationdata after a short period of time (e.g., 24 hours or 3 days). Currentconsent from the participants may be required for each access of thestored video communication data by another participant. Alternatively,in some embodiments, participant devices may be able to stream or sendstored video communication data that may only be stored at the oneparticipant device. For example, each participant device may retain onlythat the video communication data captured at the participant device,bypassing any remote storage at all for the video communication.

FIG. 17 is a block diagram illustrating a connection manager of a videocommunication module implementing dynamic transitions between videocommunications and video messages, according to some embodiments.Connection manager 320 may implement a video message transitioncoordinator 1720, which may be configured to direct the various stepsnecessary to transition between a video message and a videocommunication. For instance, start or stop commands 1704 for videocommunications may be sent to transmission/reception manager 330, tostart or stop the video communication. Similarly, video messagetransition coordinator may request that another application, such as anapplication, module or component responsible for implementing videomessaging (e.g., instant messaging module 3041 in portablemulti-function device 3000 in FIG. 22) to handle video communicationdata as a video message (e.g., such as in the event video connectionquality falls below a threshold). Video message transaction coordinatormay determine when transitions between video messaging and videocommunication occur based on various inputs. For example, an indicationor a user selection 1706 to initiate a video communication for acurrently displayed video message may be received, which video messagetransition coordinator may rely upon to initiate a transition to a videocommunication.

Connection manager 320 may also implement connection quality monitor1710 which may be configured to perform the various techniques describedabove with regard to FIG. 15, in order to evaluate connection qualitydata 1702 and determine when transition to and from video messaging maybe performed. Connection quality monitor 1710 may also provide input tovideo message transaction coordinator to indicate when such transitionsmay need to be performed.

Connection manger 320 may also implement video connection interface1730. Video connection interface 1730 may establish connections 1708with remote mobile computing devices in order to transition to videocommunications. Connection information 1708 may be provided totransmission/reception manager 330, or other similar componentresponsible for transmitting capture video communication data via theestablished connection(s) 1708.

Transmission reception manager 330 may record/store video communicationdata that is transmitted and/or received 1732 in stored video data 360.Stored video data 360 may be a local persistent data store, or more maybe a remote data store (e.g., single remote device or distributedsystem) storing the video communication data at one or across multiplestorage servers or locations.

Another technique for increasing the ease and flexibility of performingvideo communications may involve reducing the amount of time it takesestablish video communications. FIG. 18 is a high-level flowchartillustrating methods and techniques for instant video communications,according to some embodiments. Instant video communications may beimplemented based on pre-established video communication connectionsthat are maintained for different participant devices. Each participantdevice may manage a list of authorized participant devices from whichinstant video communications may be received or to which instant videocommunications may be sent. Please note that the term instant may referto the transmission time of video communication data between participantdevices without having to performing certain connection establishmenttechniques prior to transmitting the video communication data. Thus an“instant” video communication may be a video communication connectionfor which previously performed connection establishments steps have beenperformed so that transmission of video communication data may begin(from the perspective of users of participant devices) “instantly,” invarious embodiments, without performing steps to establish theconnection. Therefore the term instant may not be construed as requiringabsolutely instantaneous communication.

As indicated at 1810, a request may be received to initiate an instantvideo communication with a remote mobile computing device. For example,some remote mobile computing devices for which instant communication isenabled/authorized may by default initiate a request for an instantvideo communication when a video communication for one of those mobilecomputing devices is requested. In response to receipt of the request, adetermination may be made as to whether the instant connection isauthorized for the particular user, as indicated at 1820. For example, alist, other organizational scheme may be applied to stored participantconnection information, with certain participants designated asauthorized for instant connections. If the participant device is on theauthorization list, then the instant video connection may be consideredauthorized.

If authorized, as indicated by the positive exit from 1820, storedconnection information for the remote mobile computing device may beaccessed, as indicated at 1830. This stored connection information maybe maintained or updated in various embodiments to ensure that theconnection with the remote mobile computing device is successfullymaintained, in various embodiments. For example, a heartbeat or otherpolling technique may periodically or aperiodically send confirmation orreauthorization requests to the remote mobile computing device to obtainupdated connection information or verify connection information. In someembodiments, the remote mobile computing device may proactively sendupdates to participant devices for which it maintains authorization forinstant video communications to update or verify connection information.The maintained connection information may be various identifiers, keys,credentials, tokens, addresses, or any other information necessary totransmit video communication data via the video communication connectionthat has been previously authorized and established.

As indicated at 1840, video communication data captured for the instantvideo communication may be transmitted according to the stored instantconnection for the remote mobile computing device, in variousembodiments. In this way, the communication data may be quickly receivedand displayed at the remote mobile computing device, providing a nearreal-time experience of video communication for users of the mobilecomputing device. FIG. 20 described below provides further detailconcerning displaying video communication data for instant videocommunications. If an instant connection is not authorized, as indicatedby the negative exit from 1820, then a video communication connectionmay first be established with the remote computing device, as indicatedat 1850. For example, various identification and authenticationexchanges both with the remote mobile computing device and/or otherthird-party systems, such as credentialing or identification systems, tofacilitate video communication with the remote mobile computing may beperformed. After establishing the video communication connection, thenvideo communication data captured for the video communication may betransmitted according to the established video communication connection,as indicated at 1860.

FIGS. 19A and 19B are high-level flow charts illustrating methods andtechniques for obtaining authorization to establish instant videocommunication connections, according to some embodiments. FIG. 19Aillustrates establishing instant video communication authorization froma requesting participant device's perspective. As indicated at 1910, arequest may be sent to a remote mobile computing device forauthorization to establish an instant video communication connection.The request may provide various identification credentials of therequesting device, such as a user identifier, contact identifier orother information. As indicated at 1920, in response to receivingauthorization, the instant video communication connection may beestablished with the remote mobile computing device, in variousembodiments. For example, various identification and authenticationexchanges both with the remote mobile computing device and/or otherthird-party systems, such as credentialing or identification systems, tofacilitate video communication with the remote mobile computing may beperformed. Once established, connection information for the instantvideo communication connection may be stored, as indicated at 1930,which may be subsequently accessed when initiating an instant videocommunication with the remote mobile computing device.

FIG. 19B illustrates techniques for establishing instant videocommunication connections from the perspective of remote mobilecomputing device or participant device's perspective that confirms ordenies authorization for instant video communication. As indicated at1940, a request may be received from a remote mobile computing devicefor authorization to establish an instant video communicationconnection. As noted above, the request may include variousidentification information. Based on this identification information, itmay be determined whether authorization for the instant videocommunication connection is received, as indicated at 1950. For example,an indication or notification may be directed toward a user of theremote mobile computing device (e.g., displaying a notification orsending an email) identifying the requesting participant device (or userof the participant device) and requesting authorization to confirm ordeny the instant connection authorization request. Confirmation inputmay be received, as indicated by the positive exit from 1950, and aresponse may be sent to the remote mobile computing device withconnection information to establish the instant video communicationconnection, as indicated at 1960. In some embodiments, the receivingparticipant device may itself begin the establishment process for theinstant video communication connection. If not authorized, then asindicated at 1970, the request for authorization may be denied.

FIG. 20 is a high-level flow chart illustrating methods and techniquesfor displaying video communication data via an instant videocommunication connection, according to some embodiments. As indicated at2010 video communication data for an instant video communication may bereceived from a remote mobile computing device. Prior to displaying thevideo communication data, it may be determined whether instant videocommunication is authorized from this remote mobile computing device, asindicate at 2020. For instance, an authorized user/device list orcontact information may be evaluated with respect to the particularremote mobile computing device. If the instant video communication isauthorized, then video communication data for the instant videocommunication may be displayed, as indicated at 2030. In someembodiments, the video communication data may be displayed by default inintegration display mode. While in some embodiments, full-screen displaymode may be used to display the video communication data, such that thedisplay location is dynamically determined. If the instant videocommunication is not authorized, then the instant video communicationmay be denied, as indicate at 2040. A denial may trigger a non-instantvideo communication establishment procedure at either remote mobilecomputing device and/or the recipient device, in some embodiments.

FIG. 21 is a block diagram illustrating a connection manager of a videocommunication module that implements instant video communications,according to some embodiments. Connection manager 320 may implementinstant connection manager 1740, which may be configured to coordinatethe authorization and performance of instant video communications, invarious embodiments. Instant connection authorization establishmentrequests 1743, such as discussed above with regard to FIG. 19A, may bereceived at instant connection manager 1740, which solicit and/orreceiving user authorization 1748 (or denial) for establishing instantvideo communications. For authorized instant video communications,instant connection manger 1740 may notify video communication interface1730 to establish the instant video communication connection with therequesting mobile computing device. Connection information for theestablished instant video communication connection may be maintained byinstant connection manager 1740 as part of authorized instant connectionparticipant information 1760.

Similarly, when user authorization 1748 (or request for) instantconnection to be established with a particular user or mobile computingdevice is received at instant connection manager 1740 (for which arequest from that particular user or participant device has not beenreceived), then instant connection manager 1740 may direct videocommunication connection interface 1730 to request authorization fromthe particular user or remote mobile computing device. Instantconnection acceptance(s) 1746 may be received for these requests, andsubsequently obtained connection information maintained as part ofauthorized instant connect participant information 1760 (which may bemaintained in participant contact information store 350 described abovewith regard to FIG. 3).

For requests to initiate an instant connection 1744, instant connectionmanager 1740 may determine whether such instant connections areauthorized, and if so, direct video communication connection interface1730 to provide the instant connection information 1752 totransmission/reception manger 330. Likewise for incoming instant videocommunications, instant connection manager 1740 may determine whetherthe incoming instant video communications are authorized, and directdisplay manager 340 to display the instant video communication data1754.

The methods described herein may in various embodiments be implementedby any combination of hardware and software. For example, in oneembodiment, the methods may be implemented by a mobile computing device,portable multi-function device (e.g., portable multi-function device3000 in FIGS. 22-24) or computer system (e.g., a computer system 4000 inFIG. 25) that includes one or more processors executing programinstructions stored on a computer-readable storage medium coupled to theprocessors. The program instructions may be configured to implement thefunctionality described herein (e.g., the functionality of variouscomponents that implement the video communication application describedherein). The various methods as illustrated in the figures and describedherein represent example embodiments of methods. The order of any methodmay be changed, and various elements may be added, reordered, combined,omitted, modified, etc.

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as PDA and/ormusic player functions. Other portable electronic devices, such aslaptops or tablet computers with touch-sensitive surfaces (e.g., touchscreen displays and/or touch pads), may also be used. It should also beunderstood that, in some embodiments, the device is not a portablecommunications device, but is a desktop computer with a touch-sensitivesurface (e.g., a touch screen display and/or a touch pad).

In the discussion that follows, an electronic device that includes adisplay and a touch-sensitive surface is described. It should beunderstood, however, that the electronic device may include one or moreother physical user-interface devices, such as a physical keyboard, amouse and/or a joystick.

The device typically supports a variety of applications, such as one ormore of the following: a drawing application, a presentationapplication, a word processing application, a website creationapplication, a disk authoring application, a spreadsheet application, agaming application, a telephone application, a video conferencingapplication, an e-mail application, an instant messaging application, aworkout support application, a photo management application, a digitalcamera application, a digital video camera application, a web browsingapplication, a digital music player application, and/or a digital videoplayer application.

The various applications that may be executed on the device may use atleast one common physical user-interface device, such as thetouch-sensitive surface. One or more functions of the touch-sensitivesurface as well as corresponding information displayed on the device maybe adjusted and/or varied from one application to the next and/or withina respective application. In this way, a common physical architecture(such as the touch-sensitive surface) of the device may support thevariety of applications with user interfaces that are intuitive andtransparent to the user.

Attention is now directed toward embodiments of portable devices withtouch-sensitive displays. FIG. 22 is a block diagram illustratingportable multifunction device 3000 with touch-sensitive displays 3012 inaccordance with some embodiments. Touch-sensitive display 3012 issometimes called a “touch screen” for convenience, and may also be knownas or called a touch-sensitive display system. Device 3000 may includememory 3002 (which may include one or more computer readable storagemediums), memory controller 3022, one or more processing units (CPU's)3020, peripherals interface 3018, RF circuitry 3008, audio circuitry3010, speaker 3011, microphone 3013, input/output (I/O) subsystem 3006,other input or control devices 3016, and external port 3024. Device 3000may include one or more optical sensors 3064. These components maycommunicate over one or more communication buses or signal lines 3003.

It should be appreciated that device 3000 is only one example of aportable multifunction device, and that device 3000 may have more orfewer components than shown, may combine two or more components, or mayhave a different configuration or arrangement of the components. Thevarious components shown in FIG. 22 may be implemented in hardware,software, or a combination of both hardware and software, including oneor more signal processing and/or application specific integratedcircuits.

Memory 3002 may include high-speed random access memory and may alsoinclude non-volatile memory, such as one or more magnetic disk storagedevices, flash memory devices, or other non-volatile solid-state memorydevices. Access to memory 3002 by other components of device 3000, suchas CPU 3020 and the peripherals interface 3018, may be controlled bymemory controller 3022.

Peripherals interface 3018 can be used to couple input and outputperipherals of the device to CPU 3020 and memory 3002. The one or moreprocessors 3020 run or execute various software programs and/or sets ofinstructions stored in memory 3002 to perform various functions fordevice 3000 and to process data.

In some embodiments, peripherals interface 3018, CPU 3020, and memorycontroller 3022 may be implemented on a single chip, such as chip 3004.In some other embodiments, they may be implemented on separate chips.

RF (radio frequency) circuitry 3008 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 3008 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 3008 may include well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 3008 may communicate with networks, such as the Internet, alsoreferred to as the World Wide Web (WWW), an intranet and/or a wirelessnetwork, such as a cellular telephone network, a wireless local areanetwork (LAN) and/or a metropolitan area network (MAN), and otherdevices by wireless communication. The wireless communication may useany of multiple communications standards, protocols and technologies,including but not limited to Global System for Mobile Communications(GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packetaccess (HSDPA), high-speed uplink packet access (HSUPA), wideband codedivision multiple access (W-CDMA), code division multiple access (CDMA),time division multiple access (TDMA), Bluetooth, Wireless Fidelity(Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and/or IEEE802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol fore-mail (e.g., Internet message access protocol (IMAP) and/or post officeprotocol (POP)), instant messaging (e.g., extensible messaging andpresence protocol (XMPP), Session Initiation Protocol for InstantMessaging and Presence Leveraging Extensions (SIMPLE), Instant Messagingand Presence Service (IMPS)), and/or Short Message Service (SMS), or anyother suitable communication protocol, including communication protocolsnot yet developed as of the filing date of this document.

Audio circuitry 3010, speaker 3011, and microphone 3013 provide an audiointerface between a user and device 3000. Audio circuitry 3010 receivesaudio data from peripherals interface 3018, converts the audio data toan electrical signal, and transmits the electrical signal to speaker3011. Speaker 3011 converts the electrical signal to human-audible soundwaves. Audio circuitry 3010 also receives electrical signals convertedby microphone 3013 from sound waves. Audio circuitry 3010 converts theelectrical signal to audio data and transmits the audio data toperipherals interface 3018 for processing. Audio data may be retrievedfrom and/or transmitted to memory 3002 and/or RF circuitry 3008 byperipherals interface 3018. In some embodiments, audio circuitry 3010also includes a headset jack (e.g., 3212, FIG. 24). The headset jackprovides an interface between audio circuitry 3010 and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

I/O subsystem 3006 couples input/output peripherals on device 3000, suchas touch screen 3012 and other input control devices 3016, toperipherals interface 3018. I/O subsystem 3006 may include displaycontroller 3056 and one or more input controllers 3060 for other inputor control devices. The one or more input controllers 3060 receive/sendelectrical signals from/to other input or control devices 3016. Theother input control devices 3016 may include physical buttons (e.g.,push buttons, rocker buttons, etc.), dials, slider switches, joysticks,click wheels, and so forth. In some alternate embodiments, inputcontroller(s) 3060 may be coupled to any (or none) of the following: akeyboard, infrared port, USB port, and a pointer device such as a mouse.The one or more buttons (e.g., 3208, FIG. 24) may include an up/downbutton for volume control of speaker 3011 and/or microphone 3013. Theone or more buttons may include a push button (e.g., 3206, FIG. 22).

Touch-sensitive display 3012 provides an input interface and an outputinterface between the device and a user. Display controller 3056receives and/or sends electrical signals from/to touch screen 3012.Touch screen 3012 displays visual output to the user. The visual outputmay include graphics, text, icons, video, and any combination thereof(collectively termed “graphics”). In some embodiments, some or all ofthe visual output may correspond to user-interface objects.

Touch screen 3012 has a touch-sensitive surface, sensor or set ofsensors that accepts input from the user based on haptic and/or tactilecontact. Touch screen 3012 and display controller 3056 (along with anyassociated modules and/or sets of instructions in memory 3002) detectcontact (and any movement or breaking of the contact) on touch screen3012 and converts the detected contact into interaction withuser-interface objects (e.g., one or more soft keys, icons, web pages orimages) that are displayed on touch screen 3012. In an exemplaryembodiment, a point of contact between touch screen 3012 and the usercorresponds to a finger of the user.

Touch screen 3012 may use LCD (liquid crystal display) technology, LPD(light emitting polymer display) technology, or LED (light emittingdiode) technology, although other display technologies may be used inother embodiments. Touch screen 3012 and display controller 3056 maydetect contact and any movement or breaking thereof using any ofmultiple touch sensing technologies now known or later developed,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies, as well as other proximity sensorarrays or other elements for determining one or more points of contactwith touch screen 3012.

Touch screen 3012 may have a video resolution in excess of 100 dpi. Insome embodiments, the touch screen has a video resolution ofapproximately 1060 dpi. The user may make contact with touch screen 3012using any suitable object or appendage, such as a stylus, a finger, andso forth. In some embodiments, the user interface is designed to workprimarily with finger-based contacts and gestures, which can be lessprecise than stylus-based input due to the larger area of contact of afinger on the touch screen. In some embodiments, the device translatesthe rough finger-based input into a precise pointer/cursor position orcommand for performing the actions desired by the user.

In some embodiments, in addition to the touch screen, device 3000 mayinclude a touchpad (not shown) for activating or deactivating particularfunctions. In some embodiments, the touchpad is a touch-sensitive areaof the device that, unlike the touch screen, does not display visualoutput. The touchpad may be a touch-sensitive surface that is separatefrom touch screen 3012 or an extension of the touch-sensitive surfaceformed by the touch screen.

Device 3000 also includes power system 3062 for powering the variouscomponents. Power system 3062 may include a power management system, oneor more power sources (e.g., battery, alternating current (AC)), arecharging system, a power failure detection circuit, a power converteror inverter, a power status indicator (e.g., a light-emitting diode(LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 3000 may also include one or more optical sensors 3064. FIG. 22shows an optical sensor coupled to optical sensor controller 3058 in I/Osubsystem 3006. Optical sensor 3064 may include charge-coupled device(CCD) or complementary metal-oxide semiconductor (CMOS)phototransistors. Optical sensor 3064 receives light from theenvironment, projected through one or more lens, and converts the lightto data representing an image. In conjunction with imaging module 3043(also called a camera module), optical sensor 3064 may capture stillimages or video. In some embodiments, an optical sensor is located onthe back of device 3000, opposite touch screen display 3012 on the frontof the device, so that the touch screen display may be used as aviewfinder for still and/or video image acquisition. In someembodiments, another optical sensor is located on the front of thedevice so that the user's image may be obtained for videoconferencingwhile the user views the other video conference participants on thetouch screen display.

Device 3000 may also include one or more proximity sensors 3066. FIG. 7shows proximity sensor 3066 coupled to peripherals interface 3018.Alternately, proximity sensor 3066 may be coupled to input controller3060 in I/O subsystem 3006. In some embodiments, the proximity sensorturns off and disables touch screen 3012 when the multifunction deviceis placed near the user's ear (e.g., when the user is making a phonecall).

Device 3000 includes one or more orientation sensors 3068. In someembodiments, the one or more orientation sensors include one or moreaccelerometers (e.g., one or more linear accelerometers and/or one ormore rotational accelerometers). In some embodiments, the one or moreorientation sensors include one or more gyroscopes. In some embodiments,the one or more orientation sensors include one or more magnetometers.In some embodiments, the one or more orientation sensors include one ormore of global positioning system (GPS), Global Navigation SatelliteSystem (GLONASS), and/or other global navigation system receivers. TheGPS, GLONASS, and/or other global navigation system receivers may beused for obtaining information concerning the location and orientation(e.g., portrait or landscape) of device 3000. In some embodiments, theone or more orientation sensors include any combination oforientation/rotation sensors. FIG. 22 shows the one or more orientationsensors 3068 coupled to peripherals interface 3018. Alternately, the oneor more orientation sensors 3068 may be coupled to an input controller3060 in I/O subsystem 3006. In some embodiments, information isdisplayed on the touch screen display in a portrait view or a landscapeview based on an analysis of data received from the one or moreorientation sensors.

In some embodiments, the software components stored in memory 3002include operating system 3026, communication module (or set ofinstructions) 3028, contact/motion module (or set of instructions) 3030,graphics module (or set of instructions) 3032, text input module (or setof instructions) 3034, Global Positioning System (GPS) module (or set ofinstructions) 3035, and applications (or sets of instructions) 3036.Furthermore, in some embodiments memory 3002 stores device/globalinternal state 3057, as shown in FIG. 22. Device/global internal state3057 includes one or more of: active application state, indicating whichapplications, if any, are currently active; display state, indicatingwhat applications, views or other information occupy various regions oftouch screen display 3012; sensor state, including information obtainedfrom the device's various sensors and input control devices 3016; andlocation information concerning the device's location and/or attitude.

Operating system 3026 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS,or an embedded operating system such as VxWorks) includes varioussoftware components and/or drivers for controlling and managing generalsystem tasks (e.g., memory management, storage device control, powermanagement, etc.) and facilitates communication between various hardwareand software components.

Communication module 3028 facilitates communication with other devicesover one or more external ports 3024 and also includes various softwarecomponents for handling data received by RF circuitry 3008 and/orexternal port 3024. External port 3024 (e.g., Universal Serial Bus(USB), FIREWIRE, etc.) is adapted for coupling directly to other devicesor indirectly over a network (e.g., the Internet, wireless LAN, etc.).In some embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with the30-pin connector used on iPod (trademark of Apple Inc.) devices.

Contact/motion module 3030 may detect contact with touch screen 3012 (inconjunction with display controller 3056) and other touch sensitivedevices (e.g., a touchpad or physical click wheel). Contact/motionmodule 3030 includes various software components for performing variousoperations related to detection of contact, such as determining ifcontact has occurred (e.g., detecting a finger-down event), determiningif there is movement of the contact and tracking the movement across thetouch-sensitive surface (e.g., detecting one or more finger-draggingevents), and determining if the contact has ceased (e.g., detecting afinger-up event or a break in contact). Contact/motion module 3030receives contact data from the touch-sensitive surface. Determiningmovement of the point of contact, which is represented by a series ofcontact data, may include determining speed (magnitude), velocity(magnitude and direction), and/or an acceleration (a change in magnitudeand/or direction) of the point of contact. These operations may beapplied to single contacts (e.g., one finger contacts) or to multiplesimultaneous contacts (e.g., “multitouch”/multiple finger contacts). Insome embodiments, contact/motion module 3030 and display controller 3056detect contact on a touchpad.

Contact/motion module 3030 may detect a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns. Thus, a gesture may be detected by detecting a particularcontact pattern. For example, detecting a finger tap gesture includesdetecting a finger-down event followed by detecting a finger-up (liftoff) event at the same position (or substantially the same position) asthe finger-down event (e.g., at the position of an icon). As anotherexample, detecting a finger swipe gesture on the touch-sensitive surfaceincludes detecting a finger-down event followed by detecting one or morefinger-dragging events, and subsequently followed by detecting afinger-up (lift off) event.

Graphics module 3032 includes various known software components forrendering and displaying graphics on touch screen 3012 or other display,including components for changing the intensity of graphics that aredisplayed. As used herein, the term “graphics” includes any object thatcan be displayed to a user, including without limitation text, webpages, icons (such as user-interface objects including soft keys),digital images, videos, animations and the like.

In some embodiments, graphics module 3032 stores data representinggraphics to be used. Each graphic may be assigned a corresponding code.Graphics module 3032 receives, from applications etc., one or more codesspecifying graphics to be displayed along with, if necessary, coordinatedata and other graphic property data, and then generates screen imagedata to output to display controller 3056.

Text input module 3034, which may be a component of graphics module3032, provides soft keyboards for entering text in various applications(e.g., contacts 3037, e-mail 3040, IM 3041, browser 3047, and any otherapplication that needs text input).

GPS module 3035 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 3038 foruse in location-based dialing, to camera 3043 as picture/video metadata,and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Applications 3036 may include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   contacts module 3037 (sometimes called an address book or        contact list);    -   telephone module 3038;    -   video communication module 3039;    -   e-mail client module 3040;    -   instant messaging (IM) module 3041;    -   workout support module 3042;    -   camera module 3043 for still and/or video images;    -   image management module 3044;    -   browser module 3047;    -   calendar module 3048;    -   widget modules 3049, which may include one or more of: weather        widget 3049-1, stocks widget 3049-2, calculator widget 3049-3,        alarm clock widget 3049-4, dictionary widget 3049-5, and other        widgets obtained by the user, as well as user-created widgets        3049-6;    -   widget creator module 3050 for making user-created widgets        3049-6;    -   search module 3051;    -   video and music player module 3052, which may be made up of a        video player    -   module and a music player module;    -   notes module 3053;    -   map module 3054; and/or    -   online video module 3055.

Examples of other applications 3036 that may be stored in memory 3002include other word processing applications, other image editingapplications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch screen 3012, display controller 3056, contactmodule 3030, graphics module 3032, and text input module 3034, contactsmodule 3037 may be used to manage an address book or contact list (e.g.,stored in application internal state 3092 of contacts module 3037 inmemory 3002), including: adding name(s) to the address book; deletingname(s) from the address book; associating telephone number(s), e-mailaddress(es), physical address(es) or other information with a name;associating an image with a name; categorizing and sorting names;providing telephone numbers or e-mail addresses to initiate and/orfacilitate communications by telephone 3038, video conference 3039,e-mail 3040, or IM 3041; and so forth.

In conjunction with RF circuitry 3008, audio circuitry 3010, speaker3011, microphone 3013, touch screen 3012, display controller 3056,contact module 3030, graphics module 3032, and text input module 3034,telephone module 3038 may be used to enter a sequence of characterscorresponding to a telephone number, access one or more telephonenumbers in address book 3037, modify a telephone number that has beenentered, dial a respective telephone number, conduct a conversation anddisconnect or hang up when the conversation is completed. As notedabove, the wireless communication may use any of multiple communicationsstandards, protocols and technologies.

In conjunction with RF circuitry 3008, audio circuitry 3010, speaker3011, microphone 3013, touch screen 3012, display controller 3056,optical sensor 3064, optical sensor controller 3058, contact module3030, graphics module 3032, text input module 3034, contact list 3037,and telephone module 3038, video communication module 3039 includesexecutable instructions to initiate, conduct, and terminate a videocommunication between a user and one or more other participants inaccordance with user instructions, such as described above with regardto FIGS. 1-21.

In conjunction with RF circuitry 3008, touch screen 3012, displaycontroller 3056, contact module 3030, graphics module 3032, and textinput module 3034, e-mail client module 3040 includes executableinstructions to create, send, receive, and manage e-mail in response touser instructions. In conjunction with image management module 3044,e-mail client module 3040 makes it very easy to create and send e-mailswith still or video images taken with camera module 3043.

In conjunction with RF circuitry 3008, touch screen 3012, displaycontroller 3056, contact module 3030, graphics module 3032, and textinput module 3034, the instant messaging module 3041 includes executableinstructions to enter a sequence of characters corresponding to aninstant message, to modify previously entered characters, to transmit arespective instant message (for example, using a Short Message Service(SMS) or Multimedia Message Service (MMS) protocol for telephony-basedinstant messages or using XMPP, SIMPLE, or IMPS for Internet-basedinstant messages), to receive instant messages and to view receivedinstant messages. In some embodiments, transmitted and/or receivedinstant messages may include graphics, photos, audio files, video filesand/or other attachments as are supported in a MMS and/or an EnhancedMessaging Service (EMS). As used herein, “instant messaging” refers toboth telephony-based messages (e.g., messages sent using SMS or MMS) andInternet-based messages (e.g., messages sent using XMPP, SIMPLE, orIMPS).

In conjunction with RF circuitry 3008, touch screen 3012, displaycontroller 3056, contact module 3030, graphics module 3032, text inputmodule 3034, GPS module 3035, map module 3054, and music player module3046, workout support module 3042 includes executable instructions tocreate workouts (e.g., with time, distance, and/or calorie burninggoals); communicate with workout sensors (sports devices); receiveworkout sensor data; calibrate sensors used to monitor a workout; selectand play music for a workout; and display, store and transmit workoutdata.

In conjunction with touch screen 3012, display controller 3056, opticalsensor(s) 3064, optical sensor controller 3058, contact module 3030,graphics module 3032, and image management module 3044, camera module3043 includes executable instructions to capture still images or video(including a video stream) and store them into memory 3002, modifycharacteristics of a still image or video, or delete a still image orvideo from memory 3002.

In conjunction with touch screen 3012, display controller 3056, contactmodule 3030, graphics module 3032, text input module 3034, and cameramodule 3043, image management module 3044 includes executableinstructions to arrange, modify (e.g., edit), or otherwise manipulate,label, delete, present (e.g., in a digital slide show or album), andstore still and/or video images.

In conjunction with RF circuitry 3008, touch screen 3012, display systemcontroller 3056, contact module 3030, graphics module 3032, and textinput module 3034, browser module 3047 includes executable instructionsto browse the Internet in accordance with user instructions, includingsearching, linking to, receiving, and displaying web pages or portionsthereof, as well as attachments and other files linked to web pages.

In conjunction with RF circuitry 3008, touch screen 3012, display systemcontroller 3056, contact module 3030, graphics module 3032, text inputmodule 3034, e-mail client module 3040, and browser module 3047,calendar module 3048 includes executable instructions to create,display, modify, and store calendars and data associated with calendars(e.g., calendar entries, to do lists, etc.) in accordance with userinstructions.

In conjunction with RF circuitry 3008, touch screen 3012, display systemcontroller 3056, contact module 3030, graphics module 3032, text inputmodule 3034, and browser module 3047, widget modules 3049 aremini-applications that may be downloaded and used by a user (e.g.,weather widget 3049-1, stocks widget 3049-2, calculator widget 3049-3,alarm clock widget 3049-4, and dictionary widget 3049-5) or created bythe user (e.g., user-created widget 3049-6). In some embodiments, awidget includes an HTML (Hypertext Markup Language) file, a CSS(Cascading Style Sheets) file, and a JavaScript file. In someembodiments, a widget includes an XML (Extensible Markup Language) fileand a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with RF circuitry 3008, touch screen 3012, display systemcontroller 3056, contact module 3030, graphics module 3032, text inputmodule 3034, and browser module 3047, the widget creator module 3050 maybe used by a user to create widgets (e.g., turning a user-specifiedportion of a web page into a widget).

In conjunction with touch screen 3012, display system controller 3056,contact module 3030, graphics module 3032, and text input module 3034,search module 3051 includes executable instructions to search for text,music, sound, image, video, and/or other files in memory 3002 that matchone or more search criteria (e.g., one or more user-specified searchterms) in accordance with user instructions.

In conjunction with touch screen 3012, display system controller 3056,contact module 3030, graphics module 3032, audio circuitry 3010, speaker3011, RF circuitry 3008, and browser module 3047, video and music playermodule 3052 includes executable instructions that allow the user todownload and play back recorded music and other sound files stored inone or more file formats, such as MP3 or AAC files, and executableinstructions to display, present or otherwise play back videos (e.g., ontouch screen 3012 or on an external, connected display via external port3024). In some embodiments, device 3000 may include the functionality ofan MP3 player, such as an iPod (trademark of Apple Inc.).

In conjunction with touch screen 3012, display controller 3056, contactmodule 3030, graphics module 3032, and text input module 3034, notesmodule 3053 includes executable instructions to create and manage notes,to do lists, and the like in accordance with user instructions.

In conjunction with RF circuitry 3008, touch screen 3012, display systemcontroller 3056, contact module 3030, graphics module 3032, text inputmodule 3034, GPS module 3035, and browser module 3047, map module 3054may be used to receive, display, modify, and store maps and dataassociated with maps (e.g., driving directions; data on stores and otherpoints of interest at or near a particular location; and otherlocation-based data) in accordance with user instructions.

In conjunction with touch screen 3012, display system controller 3056,contact module 3030, graphics module 3032, audio circuitry 3010, speaker3011, RF circuitry 3008, text input module 3034, e-mail client module3040, and browser module 3047, online video module 3055 includesinstructions that allow the user to access, browse, receive (e.g., bystreaming and/or download), play back (e.g., on the touch screen or onan external, connected display via external port 3024), send an e-mailwith a link to a particular online video, and otherwise manage onlinevideos in one or more file formats, such as H.264. In some embodiments,instant messaging module 3041, rather than e-mail client module 3040, isused to send a link to a particular online video.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 3002 maystore a subset of the modules and data structures identified above.Furthermore, memory 3002 may store additional modules and datastructures not described above.

In some embodiments, device 3000 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen and/or a touchpad. By using a touch screen and/ora touchpad as the primary input control device for operation of device3000, the number of physical input control devices (such as pushbuttons, dials, and the like) on device 3000 may be reduced.

The predefined set of functions that may be performed exclusivelythrough a touch screen and/or a touchpad include navigation between userinterfaces. In some embodiments, the touchpad, when touched by the user,navigates device 3000 to a main, home, or root menu from any userinterface that may be displayed on device 3000. In such embodiments, thetouchpad may be referred to as a “menu button.” In some otherembodiments, the menu button may be a physical push button or otherphysical input control device instead of a touchpad.

FIG. 23 is a block diagram illustrating exemplary components for eventhandling in accordance with some embodiments. In some embodiments,memory 3002 (in FIG. 23) includes event sorter 3070 (e.g., in operatingsystem 3026) and a respective application 3036-1 (e.g., any of theaforementioned applications 3037-3051, 3055).

Event sorter 3070 receives event information and determines theapplication 3036-1 and application view 3091 of application 3036-1 towhich to deliver the event information. Event sorter 3070 includes eventmonitor 3071 and event dispatcher module 3074. In some embodiments,application 3036-1 includes application internal state 3092, whichindicates the current application view(s) displayed on touch sensitivedisplay 3012 when the application is active or executing. In someembodiments, device/global internal state 3057 is used by event sorter3070 to determine which application(s) is (are) currently active, andapplication internal state 3092 is used by event sorter 3070 todetermine application views 3091 to which to deliver event information,such as which applications may control display of data on display 3012and configure display attributes for video communications, as discussedabove with regard to FIGS. 1-10.

In some embodiments, application internal state 3092 includes additionalinformation, such as one or more of: resume information to be used whenapplication 3036-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 3036-1, a state queue for enabling the user to go back toa prior state or view of application 3036-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 3071 receives event information from peripherals interface3018. Event information includes information about a sub-event (e.g., auser touch on touch sensitive display 3012, as part of a multi-touchgesture). Peripherals interface 3018 transmits information it receivesfrom I/O subsystem 3006 or a sensor, such as proximity sensor 3066,orientation sensor(s) 3068, and/or microphone 3013 (through audiocircuitry 3010). Information that peripherals interface 3018 receivesfrom I/O subsystem 3006 includes information from touch-sensitivedisplay 3012 or a touch-sensitive surface.

In some embodiments, event monitor 3071 sends requests to theperipherals interface 3018 at predetermined intervals. In response,peripherals interface 3018 transmits event information. In otherembodiments, peripheral interface 3018 transmits event information onlywhen there is a significant event (e.g., receiving an input above apredetermined noise threshold and/or for more than a predeterminedduration).

In some embodiments, event sorter 3070 also includes a hit viewdetermination module 3072 and/or an active event recognizerdetermination module 3073.

Hit view determination module 3072 provides software procedures fordetermining where a sub-event has taken place within one or more views,when touch sensitive display 3012 displays more than one view. Views aremade up of controls and other elements that a user can see on thedisplay.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of a respective application) inwhich a touch is detected may correspond to programmatic levels within aprogrammatic or view hierarchy of the application. For example, thelowest level view in which a touch is detected may be called the hitview, and the set of events that are recognized as proper inputs may bedetermined based, at least in part, on the hit view of the initial touchthat begins a touch-based gesture.

Hit view determination module 3072 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 3072identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (i.e., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule, the hit view typically receives all sub-events related to thesame touch or input source for which it was identified as the hit view.

Active event recognizer determination module 3073 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 3073 determines that only the hit view should receive aparticular sequence of sub-events. In other embodiments, active eventrecognizer determination module 3073 determines that all views thatinclude the physical location of a sub-event are actively involvedviews, and therefore determines that all actively involved views shouldreceive a particular sequence of sub-events. In other embodiments, evenif touch sub-events were entirely confined to the area associated withone particular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 174 dispatches the event information to an eventrecognizer (e.g., event recognizer 3080). In embodiments includingactive event recognizer determination module 3073, event dispatchermodule 3074 delivers the event information to an event recognizerdetermined by active event recognizer determination module 3073. In someembodiments, event dispatcher module 3074 stores in an event queue theevent information, which is retrieved by a respective event receivermodule 3082.

In some embodiments, operating system 3026 includes event sorter 3070.Alternatively, application 3036-1 includes event sorter 3070. In yetother embodiments, event sorter 3070 is a stand-alone module, or a partof another module stored in memory 3002, such as contact/motion module3030.

In some embodiments, application 3036-1 includes multiple event handlers3090 and one or more application views 3091, each of which includesinstructions for handling touch events that occur within a respectiveview of the application's user interface. Each application view 3091 ofthe application 3036-1 includes one or more event recognizers 3080.Typically, a respective application view 3091 includes multiple eventrecognizers 3080. In other embodiments, one or more of event recognizers3080 are part of a separate module, such as a user interface kit (notshown) or a higher level object from which application 3036-1 inheritsmethods and other properties. In some embodiments, a respective eventhandler 3090 includes one or more of: data updater 3076, object updater3077, GUI updater 3078, and/or event data 3079 received from eventsorter 3070. Event handler 3090 may utilize or call data updater 3076,object updater 3077 or GUI updater 3078 to update the applicationinternal state 3092. Alternatively, one or more of the application views3091 includes one or more respective event handlers 3090. Also, in someembodiments, one or more of data updater 3076, object updater 3077, andGUI updater 3078 are included in a respective application view 3091.

A respective event recognizer 3080 receives event information (e.g.,event data 3079) from event sorter 3070, and identifies an event fromthe event information. Event recognizer 3080 includes event receiver3082 and event comparator 3084. In some embodiments, event recognizer3080 also includes at least a subset of: metadata 3083, and eventdelivery instructions 3088 (which may include sub-event deliveryinstructions).

Event receiver 3082 receives event information from event sorter 3070.The event information includes information about a sub-event, forexample, a touch or a touch movement. Depending on the sub-event, theevent information also includes additional information, such as locationof the sub-event. When the sub-event concerns motion of a touch theevent information may also include speed and direction of the sub-event.In some embodiments, events include rotation of the device from oneorientation to another (e.g., from a portrait orientation to a landscapeorientation, or vice versa), and the event information includescorresponding information about the current orientation (also calleddevice attitude) of the device.

Event comparator 3084 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 3084 includes eventdefinitions 3086. Event definitions 3086 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(3087-1), event 2 (3087-2), and others. In some embodiments, sub-eventsin an event 3087 include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (3087-1) is a double tap on a displayed object.The double tap, for example, includes a first touch (touch begin) on thedisplayed object for a predetermined phase, a first lift-off (touch end)for a predetermined phase, a second touch (touch begin) on the displayedobject for a predetermined phase, and a second lift-off (touch end) fora predetermined phase. In another example, the definition for event 2(1087-2) is a dragging on a displayed object. The dragging, for example,includes a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across touch-sensitivedisplay 3012, and lift-off of the touch (touch end). In someembodiments, the event also includes information for one or moreassociated event handlers 3090.

In some embodiments, event definition 3087 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 3084 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed ontouch-sensitive display 3012, when a touch is detected ontouch-sensitive display 3012, event comparator 3084 performs a hit testto determine which of the three user-interface objects is associatedwith the touch (sub-event). If each displayed object is associated witha respective event handler 3090, the event comparator uses the result ofthe hit test to determine which event handler 3090 should be activated.For example, event comparator 3084 selects an event handler associatedwith the sub-event and the object triggering the hit test.

In some embodiments, the definition for a respective event 3087 alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 3080 determines that the series ofsub-events do not match any of the events in event definitions 3086, therespective event recognizer 3080 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 3080 includesmetadata 3083 with configurable properties, flags, and/or lists thatindicate how the event delivery system should perform sub-event deliveryto actively involved event recognizers. In some embodiments, metadata3083 includes configurable properties, flags, and/or lists that indicatehow event recognizers may interact with one another. In someembodiments, metadata 3083 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 3080 activates eventhandler 3090 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 3080 delivers event information associated with theevent to event handler 3090. Activating an event handler 3090 isdistinct from sending (and deferred sending) sub-events to a respectivehit view. In some embodiments, event recognizer 3080 throws a flagassociated with the recognized event, and event handler 3090 associatedwith the flag catches the flag and performs a predefined process.

In some embodiments, event delivery instructions 3088 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 3076 creates and updates data used inapplication 3036-1. For example, data updater 3076 updates the telephonenumber used in contacts module 3037, or stores a video file used invideo player module 3045. In some embodiments, object updater 3077creates and updates objects used in application 3036-1. For example,object updater 3076 creates a new user-interface object or updates theposition of a user-interface object. GUI updater 3078 updates the GUI.For example, GUI updater 3078 prepares display information and sends itto graphics module 3032 for display on a touch-sensitive display.

In some embodiments, event handler(s) 3090 includes or has access todata updater 3076, object updater 3077, and GUI updater 3078. In someembodiments, data updater 3076, object updater 3077, and GUI updater3078 are included in a single module of a respective application 3036-1or application view 3091. In other embodiments, they are included in twoor more software modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate multifunction devices 3000 withinput-devices, not all of which are initiated on touch screens, e.g.,coordinating mouse movement and mouse button presses with or withoutsingle or multiple keyboard presses or holds, user movements taps,drags, scrolls, etc., on touch-pads, pen stylus inputs, movement of thedevice, oral instructions, detected eye movements, biometric inputs,and/or any combination thereof, which may be utilized as inputscorresponding to sub-events which define an event to be recognized.

FIG. 24 illustrates a portable multifunction device 3000 having a touchscreen 3012 in accordance with some embodiments. The touch screen maydisplay one or more graphics within user interface (UI) 3200. In thisembodiment, as well as others described below, a user may select one ormore of the graphics by making a gesture on the graphics, for example,with one or more fingers 3202 (not drawn to scale in the figure) or oneor more styluses 3203 (not drawn to scale in the figure). In someembodiments, selection of one or more graphics occurs when the userbreaks contact with the one or more graphics. In some embodiments, thegesture may include one or more taps, one or more swipes (from left toright, right to left, upward and/or downward) and/or a rolling of afinger (from right to left, left to right, upward and/or downward) thathas made contact with device 3000. In some embodiments, inadvertentcontact with a graphic may not select the graphic. For example, a swipegesture that sweeps over an application icon may not select thecorresponding application when the gesture corresponding to selection isa tap.

Device 3000 may also include one or more physical buttons, such as“home” or menu button 3204. As described previously, menu button 3204may be used to navigate to any application 3036 in a set of applicationsthat may be executed on device 3000. Alternatively, in some embodiments,the menu button is implemented as a soft key in a GUI displayed on touchscreen 3012.

In one embodiment, device 3000 includes touch screen 3012, menu button3204, push button 3206 for powering the device on/off and locking thedevice, volume adjustment button(s) 3208, Subscriber Identity Module(SIM) card slot 3210, head set jack 3212, and docking/charging externalport 3024. Push button 3206 may be used to turn the power on/off on thedevice by depressing the button and holding the button in the depressedstate for a predefined time interval; to lock the device by depressingthe button and releasing the button before the predefined time intervalhas elapsed; and/or to unlock the device or initiate an unlock process.In an alternative embodiment, device 3000 also may accept verbal inputfor activation or deactivation of some functions through microphone3013.

It should be noted that, although many of the following examples will begiven with reference to inputs on touch screen 3012 (where the touchsensitive surface and the display are combined), a touch-sensitivesurface that is separate from the display may be used instead of touchscreen 3012.

Embodiments of the various methods and techniques for videocommunication as described herein may be executed on one or morecomputer systems which may interact with various other participantdevices, such as portable multi-function computing device 3000 describedabove with regard to FIGS. 22-24. One such computer system isillustrated by FIG. 25. In different embodiments, computer system 4000may be any of various types of devices, including, but not limited to, apersonal computer system, desktop computer, laptop, notebook, or netbookcomputer, mainframe computer system, handheld computer, workstation,network computer, a camera, a set top box, a mobile device, a consumerdevice, video game console, handheld video game device, applicationserver, storage device, a peripheral device such as a switch, modem,router, or in general any type of computing or electronic device.

In the illustrated embodiment, computer system 4000 includes one or moreprocessors 4010 coupled to a system memory 4020 via an input/output(I/O) interface 4030. Computer system 4000 further includes a networkinterface 4040 coupled to I/O interface 4030, and one or moreinput/output devices 4050, such as cursor control device 4060, keyboard4070, display(s) 4080, as well as various audio/optical sensors 4090 forcapturing video and audio data. In some embodiments, it is contemplatedthat embodiments may be implemented using a single instance of computersystem 4000, while in other embodiments multiple such systems, ormultiple nodes making up computer system 4000, may be configured to hostdifferent portions or instances of embodiments.

In various embodiments, computer system 4000 may be a uniprocessorsystem including one processor 4010, or a multiprocessor systemincluding several processors 4010 (e.g., two, four, eight, or anothersuitable number). Processors 4010 may be any suitable processor capableof executing instructions. For example, in various embodiments,processors 4010 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 4010 may commonly,but not necessarily, implement the same ISA.

In some embodiments, at least one processor 4010 may be a graphicsprocessing unit. A graphics processing unit or GPU may be considered adedicated graphics-client device for a personal computer, workstation,game console or other computing or electronic device. Modern GPUs may bevery efficient at manipulating and displaying computer graphics, andtheir highly parallel structure may make them more effective thantypical CPUs for a range of complex graphical algorithms. For example, agraphics processor may implement a number of graphics primitiveoperations in a way that makes executing them much faster than drawingdirectly to the screen with a host central processing unit (CPU). Invarious embodiments, the image processing methods disclosed herein may,at least in part, be implemented by program instructions configured forexecution on one of, or parallel execution on two or more of, such GPUs.The GPU(s) may implement one or more application programmer interfaces(APIs) that permit programmers to invoke the functionality of theGPU(s). Suitable GPUs may be commercially available from vendors such asNVIDIA Corporation, ATI Technologies (AMD), and others.

System memory 4020 may be configured to store program instructionsand/or data accessible by processor 4010. In various embodiments, systemmemory 4020 may be implemented using any suitable memory technology,such as static random access memory (SRAM), synchronous dynamic RAM(SDRAM), nonvolatile/Flash-type memory, or any other type of memory. Inthe illustrated embodiment, program instructions and data implementingdesired functions, such as those described above for videocommunications as described herein are shown stored within system memory4020 as program instructions 4025 and data storage 4035, respectively.In other embodiments, program instructions and/or data may be received,sent or stored upon different types of computer-accessible media or onsimilar media separate from system memory 4020 or computer system 4000.Generally speaking, a computer-accessible medium may include storagemedia or memory media such as magnetic or optical media, e.g., disk orCD/DVD-ROM coupled to computer system 4000 via I/O interface 4030.Program instructions and data stored via a computer-accessible mediummay be transmitted by transmission media or signals such as electrical,electromagnetic, or digital signals, which may be conveyed via acommunication medium such as a network and/or a wireless link, such asmay be implemented via network interface 4040.

In one embodiment, I/O interface 4030 may be configured to coordinateI/O traffic between processor 4010, system memory 4020, and anyperipheral devices in the device, including network interface 4040 orother peripheral interfaces, such as input/output devices 4050. In someembodiments, I/O interface 4030 may perform any necessary protocol,timing or other data transformations to convert data signals from onecomponent (e.g., system memory 4020) into a format suitable for use byanother component (e.g., processor 4010). In some embodiments, I/Ointerface 4030 may include support for devices attached through varioustypes of peripheral buses, such as a variant of the Peripheral ComponentInterconnect (PCI) bus standard or the Universal Serial Bus (USB)standard, for example. In some embodiments, the function of I/Ointerface 4030 may be split into two or more separate components, suchas a north bridge and a south bridge, for example. In addition, in someembodiments some or all of the functionality of I/O interface 4030, suchas an interface to system memory 4020, may be incorporated directly intoprocessor 4010.

Network interface 4040 may be configured to allow data to be exchangedbetween computer system 4000 and other devices attached to a network,such as other computer systems, or between nodes of computer system4000. In various embodiments, network interface 4040 may supportcommunication via wired or wireless general data networks, such as anysuitable type of Ethernet network, for example; viatelecommunications/telephony networks such as analog voice networks ordigital fiber communications networks; via storage area networks such asFibre Channel SANs, or via any other suitable type of network and/orprotocol.

Input/output devices 4050 may, in some embodiments, include one or moredisplay terminals, keyboards, keypads, touchpads, scanning devices,voice or optical recognition devices, or any other devices suitable forentering or retrieving data by one or more computer system 4000.Multiple input/output devices 4050 may be present in computer system4000 or may be distributed on various nodes of computer system 4000. Insome embodiments, similar input/output devices may be separate fromcomputer system 4000 and may interact with one or more nodes of computersystem 4000 through a wired or wireless connection, such as over networkinterface 4040.

As shown in FIG. 25, memory 4020 may include program instructions 4025,configured to implement embodiments of the video communicationtechniques as described herein, and data storage 4035, comprisingvarious data accessible by program instructions 4025. Data storage 4035may include data that may be used in embodiments. In other embodiments,other or different software elements and data may be included.

Those skilled in the art will appreciate that computer system 4000 ismerely illustrative and is not intended to limit the scope of themethods and techniques as described herein. In particular, the computersystem and devices may include any combination of hardware or softwarethat can perform the indicated functions, including a computer, personalcomputer system, desktop computer, laptop, notebook, or netbookcomputer, mainframe computer system, handheld computer, workstation,network computer, a camera, a set top box, a mobile device, networkdevice, internet appliance, PDA, wireless phones, pagers, a consumerdevice, video game console, handheld video game device, applicationserver, storage device, a peripheral device such as a switch, modem,router, or in general any type of computing or electronic device.Computer system 4000 may also be connected to other devices that are notillustrated, or instead may operate as a stand-alone system. Inaddition, the functionality provided by the illustrated components mayin some embodiments be combined in fewer components or distributed inadditional components. Similarly, in some embodiments, the functionalityof some of the illustrated components may not be provided and/or otheradditional functionality may be available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management and dataintegrity. Alternatively, in other embodiments some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated computer system via inter-computercommunication. Some or all of the system components or data structuresmay also be stored (e.g., as instructions or structured data) on acomputer-accessible medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome embodiments, instructions stored on a computer-accessible mediumseparate from computer system 4000 may be transmitted to computer system4000 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link. Various embodiments mayfurther include receiving, sending or storing instructions and/or dataimplemented in accordance with the foregoing description upon acomputer-accessible medium. Accordingly, the present invention may bepracticed with other computer system configurations.

In the previous detailed description, numerous specific details are setforth to provide a thorough understanding of claimed subject matter.However, it will be understood by those skilled in the art that claimedsubject matter may be practiced without these specific details. In otherinstances, methods, apparatus, or systems that would be known by one ofordinary skill have not been described in detail so as not to obscureclaimed subject matter.

It will also be understood that, although the terms first, second, etc.may be used herein to describe various elements, these elements shouldnot be limited by these terms. These terms are only used to distinguishone element from another. For example, a first contact could be termed asecond contact, and, similarly, a second contact could be termed a firstcontact, without departing from the scope of the present invention. Thefirst contact and the second contact are both contacts, but they are notthe same contact.

The terminology used in the detailed description herein is for thepurpose of describing particular embodiments only and is not intended tobe limiting of the invention. As used in the description and theappended claims, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in response to detecting,” dependingon the context. Similarly, the phrase “if it is determined” or “if [astated condition or event] is detected” may be construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event],” depending on the context.

Some portions of the detailed description which follow are presented interms of algorithms or symbolic representation of operations on binarydigital signals stored within a memory of a specific apparatus orspecial purpose computing device or platform.

In the context of this particular specification, the term specificapparatus or the like includes a general purpose computer once it isprogrammed to perform particular functions pursuant to instructions fromprogram software and other programmable electronic devices. Algorithmicdescriptions or symbolic representations are examples of techniques usedby those of ordinary skill in the signal processing or related arts toconvey the substance of their work to others skilled in the art. Analgorithm is here, and is generally, considered to be a self-consistentsequence of operations or similar signal processing leading to a desiredresult. In this context, operations or processing involve physicalmanipulation of physical quantities. Typically, although notnecessarily, such quantities may take the form of electrical or magneticsignals capable of being stored, transferred, combined, compared orotherwise manipulated. It has proven convenient at times, principallyfor reasons of common usage, to refer to such signals as bits, data,values, elements, symbols, characters, terms, numbers, numerals or thelike. It should be understood, however, that all of these or similarterms are to be associated with appropriate physical quantities and aremerely convenient labels.

What is claimed is:
 1. A system, comprising: an electronic display; oneor more processors; a memory comprising program instructions that whenexecuted by the one or more processors cause the one or more processorsto implement: a video communication module, configured to implementvideo communications; an application module configured to: provide imagedata for at least a portion of the electronic display for theapplication module; and define one or more display attributes thatindicate a display location, relative to the at least the portion of theelectronic display for the application module, for video communicationdata from a module other than the application module, wherein theapplication module defines the one or more display attributes atruntime; an audio/video management module, configured to: receive, viathe video communication module, video communication data for display aspart of a current video communication; identify, in response to thecurrent video communication and based on the one or more displayattributes defined by the application module at runtime, a displaylocation within the electronic display for the video communication data,wherein the image data for the application module is currently displayedwithin the electronic display; and direct display of the videocommunication data in the determined display location within theelectronic display.
 2. The system of claim 1, wherein the audio/videomanagement module is further configured to display one or more userinterface elements in the determined location within the electronicdisplay.
 3. The system of claim 2, wherein the electronic display is atouch-sensitive device, and wherein the video communication module isfurther configured to: receive an indication of user input as a touchgesture via the electronic display, wherein the touch gesture initiatedwithin the determined location of the electronic display; in responsereceiving the indication of the user input: pause the current videocommunication; resume the current video communication; or end thecurrent video communication.
 4. The system of claim 1, wherein to directdisplay of the video communication data in the determined displaylocation within the electronic display, the audio/video managementmodule is configured to: integrate the video communication data withother display data obtained via the application module into combineddata to be displayed in the determined display location within theelectronic display.
 5. The system of claim 1, wherein the electronicdisplay is a touch-sensitive device, and wherein the audio/videocommunication module is further configured to: receive an indication ofuser input as a touch gesture via the electronic display, wherein thetouch gesture is initiated within the determined location of theelectronic display, wherein the user input requests a switch tofull-screen display mode for the video communication module; and inresponse to receipt of the indication of the user input, direct displayof the video communication data according to the full-screen displaymode of the video communication module within the electronic display. 6.The system of claim 1, wherein the system is a mobile phone.
 7. Amethod, comprising: performing, by a portable multi-function computingdevice: defining, by an application, one or more display attributes thatindicate a display location, relative to at least a portion of anelectronic display for the application, for video communication datafrom a second application other than the application, wherein theapplication defines the one or more display attributes at runtime;receiving, by a video communication application, video communicationdata for display as part of a current video communication; determining,based on the one or more display attributes defined by the applicationat runtime, a display location for the video communication data withinthe electronic display currently displaying image data for theapplication; and displaying the video communication data in thedetermined display location within the electronic display.
 8. The methodof claim 7, further comprising: detecting a switch in display mode forthe video communication, wherein the switch in the display mode is achange from a full-screen display mode to an integration display mode;and in response to detecting the switch in display mode to theintegration display mode, performing said determining the displaylocation and said displaying the video communication data.
 9. The methodof claim 8, further comprising detecting another switch in the displaymode for the video communication, wherein the other switch is a changefrom the integration display mode to the full-screen display mode forthe video communication application, wherein subsequently received videocommunication data for the video communication is displayed in thefull-screen display mode.
 10. The method of claim 7, further comprising:receiving an indication of user input for the current videocommunication; in response to receiving the indication of the userinput, performing one of: pausing the current video communication;resuming the current video communication; or ending the current videocommunication.
 11. The method of claim 7 further comprising: receivingan indication of user input for the current video communication; and inresponse to receiving the indication of the user input, displaying oneor more user interface elements in the determined location within theelectronic display.
 12. The method of claim 7, wherein said displayingthe video communication data in the determined display location withinthe electronic display comprises integrating the video communicationdata with other display data obtained via the application into combineddata to be displayed in the determined display location within theelectronic display.
 13. The method of claim 7, wherein the current videocommunication is a group video communication between a plurality ofdifferent computing devices including the portable multi-functioncomputing device, wherein the video communication data comprisesrespective video communication data for each of the plurality ofdifferent computing devices; and wherein said determining the displaylocation within the electronic display comprises determining a differentrespective location for the respective video communication data for eachof the plurality of different computing devices.
 14. A non-transitory,computer-readable storage medium, storing program instructions that whenexecuted by a mobile computing device causes the mobile computing deviceto implement: displaying image data within at least a portion of anelectronic display for a currently executing application, wherein thecurrently executing application is configured to define one or moredisplay attributes that indicate a display location, relative to the atleast the portion of the electronic display for the currently executingapplication, for video communication data from a second applicationother than the currently executing application, and wherein thecurrently executing application defines the one or more displayattributes at runtime; detecting initiation of a video communication viaa video communication application; in response to receiving videocommunication data for the video communication: determining, based onthe one or more display attributes defined by the currently executingapplication at runtime, a display location within the electronic displayfor the video communication data; and displaying the video communicationdata in the determined display location within the electronic display.15. The non-transitory, computer-readable storage medium of claim 14,wherein the currently executing application is another videocommunication application different than the video communicationapplication.
 16. The non-transitory, computer-readable storage medium ofclaim 14, wherein the program instructions cause the mobile computingdevice to further implement: receiving an indication of user input for acurrent video communication; in response to receiving the indication ofthe user input, performing one of: pausing the current videocommunication; resuming the current video communication; or ending thecurrent video communication.
 17. The non-transitory, computer-readablestorage medium of claim 14, wherein the program instructions cause themobile computing device to further implement: receiving an indication ofuser input to switch a display mode for the video communicationapplication to a full-screen display mode for the video communicationapplication; and in response to receiving the indication, displaying thevideo communication data according to the full-screen display mode ofthe video communication application.
 18. The non-transitory,computer-readable storage medium of claim 14, wherein, in saiddisplaying the video communication data in the determined displaylocation within the electronic display, the program instructions causethe mobile computing device to implement: integrating the videocommunication data with other display data generated by anotherapplication into combined display data to be displayed in the determineddisplay location within the electronic display.
 19. The non-transitory,computer-readable storage medium of claim 14, wherein the programinstructions further cause the mobile computing device to implement:detecting a change to the one or more display attributes by thecurrently executing application; in response to detecting the change tothe one or more display attributes: determining an updated displaylocation for the video communication data within the electronic display;and displaying the video communication data in the updated displaylocation within the electronic display.
 20. The non-transitory,computer-readable storage medium of claim 14, wherein the videocommunication is a group video communication between a plurality ofdifferent computing devices including the mobile computing device,wherein the video communication data comprises respective videocommunication data for each of the plurality of different computingdevices; and wherein said determining the display location within theelectronic display comprises determining a different respective locationfor the respective video communication data for each of the plurality ofdifferent computing devices.